Introducing Function Publishing via REST

September 14th, 2011 by Ken Elkabany

We’ve been devoting significant time to making PiCloud a useful utility outside of the Python ecosystem. The first feature we have to showcase this is the ability to publish a Python function to a URL. There are a few reasons you might want to do this:

  1. To call Python functions from a programming language other than Python. For example, if you’re integrating the PiCloud platform into a Java codebase, or even into a smartphone application (Android or iPhone).
  2. To use PiCloud from Google AppEngine, since our cloud client library is not supported on GAE.
  3. Because you’re tired of setting up web application projects when what you really need is a scalable RPC system.

In this post, we’ll give you your first taste of publishing functions on the web.

Define your Function

Just like when you offload regular computation to PiCloud, feel free to do anything in your function including importing custom libraries and making external connections.

def add(x, y):
    """This function adds!"""
    return x+y

Publish It

>>> import cloud
>>> cloud.setkey(key, secret_key)
>>> cloud.rest.publish(add, 'addition')
'https://api.picloud.com/r/2/addition'

The first argument, add, is your function. The second argument, addition, is a label so you can reference the function later; it’s also present in the returned URL for clarity. For a list of all other arguments, refer to the cloud.rest module documentation.

Let’s get information about the function we just published by making a GET request on the returned url. We recommend curl to do this from a shell. We authenticate requests using basic authentication. In curl, use “-u” as shown below to specify your key as your username, and secret key as your password. Note that we automatically extract the function’s doc string as the description.

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition/
{"output_encoding": "json", "version": "0.1", "description": "This function adds!", "signature": "addition(x, y)", "uri": "https://api.picloud.com/r/2/addition", "label": "addition"}

You can also see your published functions from your account control panel.

Call the Published Function

Now let’s call the function by using a POST request to the same URL. To specify arguments to the function add, you simply pass them in as JSON encoded POST values. In this case, you would specify the POST values x and y.

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition/ -d x=1 -d y=1
{'jid': 809730}

Get the Result

There are two ways we can grab the result of this job. The standard way is through your Python console:

>>> import cloud
>>> cloud.setkey(key, secret_key)
>>> cloud.result(809730)
2

The language-agnostic way to do this using our REST API is to query the following URL: https://api.picloud.com/job/{job_id}/result/.

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809730/result/
{"result": 2}

The difference between these two methods is that cloud.result will block until the result is ready; our REST API will return a “job not done” error, so you’ll have to keep querying until it’s ready.

For a full specification of our API, please see our REST API documentation.

Taking Advantage of JSON Arguments

Since arguments are specified as JSON, you can easily pass in strings, lists, and dictionaries into your published functions. For example, we can concatenate two strings using our addition function:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition -d "x=\"Hello, \"" -d "y=\"World\""
{'jid': 809731}
$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809731/result/
{"result": "Hello, World"}

We can also merge two lists using our addition function:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition -d "x=[1,2,3]" -d "y=[4,5,6]"
{'jid': 809732}
$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809732/result/
{"result": [1, 2, 3, 4, 5, 6]}

These work, of course, because in Python the addition operator can be applied to strings and lists, not just numbers.

Handling Raw Data

JSON does not natively support binary data. While you can encode the data to base64, and decode it in your function, we offer a more straightforward and efficient method. Binary data can be passed into a published function by using multipart/form-data as a file upload (MIME Content-Disposition sub-header has a filename parameter).

Example

To showcase raw data handling, we’re going to publish a function to create thumbnails. We’ll use this picture of Albert Einstein.



Here’s the function we’ll use to create a thumbnail of an image. We use StringIO so that we can open and save the image in a memory buffer, rather than to a file.

from PIL import Image
from cStringIO import StringIO

def thumbnail(raw_img_data, width=50, height=50, output_format='JPEG'):
    im = Image.open(StringIO(raw_img_data))
    im.thumbnail((width, height))
    out_data = StringIO()
    im.save(out_data, output_format)
    return out_data.getvalue()

import cloud
# be sure to set the output encoding to raw
cloud.rest.publish(thumbnail, 'thumbnail', out_encoding='raw')

Call the function. Use -F in conjunction with the @ symbol to POST an image file as a file upload, which will be treated as raw data by PiCloud. We can adjust the width and height by passing in POST values, or if we omit them, the default value of 50 will be used.

$ curl -k -u 'key:secret_key' -F width=60 -F height=76 -F "raw_img_data=@albert_einstein.jpg" https://api.picloud.com/r/2/thumbnail/
{'jid': 809737}

The content of the result is the binary data representing the thumbnail image. Unlike JSON encoded results, there is no enclosing dictionary. Thus, all you have to do to see the image is pipe the result of the job into a file.

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809737/result/ > albert_einstein.thumb.jpg

Open the thumbnail in your favorite image program!

Albert Einstein Thumbnail

Conclusion: Take a rest, and then give it a spin!

We’re particularly excited by function publishing because it bridges PiCloud with the world outside of Python, and in doing so, brings all the computing benefits of our standard service. You can publish functions without any care for the amount of hardware running underneath. As your functions get called more frequently, we automatically scale our servers to meet demand. You can also reserve real-time cores if they want to guarantee a certain number of cores at all times. Lastly, you can be confident that your computation is being run on a system built with performance, robustness, and redundancy at its core.

If this technology captivates you, follow us on Twitter, or go above and beyond and join our team!

Tags: , ,

Categories: What's New

You can follow any responses to this entry through the RSS 2.0 feed.

8 Responses to “Introducing Function Publishing via REST”

  1. Eugene says:

    Wow cool~

    If this functionality is available just 1 week ago, my application design would have been extremely different.

    but great work!

    this will certainly reduce the loading on servers; less need to pack the functions and arguments to PiCloud.

    Superb!

  2. It is one of the most brilliant idea and execution I have seen for a cloud service. Brilliant Stuff.

  3. Burak SARICA says:

    Really great stuff. Do the result outputs have proper MIME types? I mean is it easy to understand the response is JSON or not?

  4. Mort Canty says:

    See it in action on the GAE,

    http://ms-image-analysis.appspot.com/static/index.html

    Upload an image and then try File -> Statistics.

    Thanks again for this great addition.

  5. Tom Gleeson says:

    What would be really nice would be the ability to ‘sign’ and ‘time limit’ a REST call (a bit like AWS S3′s signed URLs). Or failing that, allow the generation of API keys with limited functionality, such as run REST queries only.

    Tom

  6. Ken Elkabany says:

    @Tom Gleeson
    If you’re hoping to grant limited access to published functions to make sharing easier, then we’re thinking along the same lines as you :)

  7. [...] Sign In Cloud Computing. Simplified. « Introducing Function Publishing via REST [...]

Leave a Reply