SlapOS Introduction

SlapOS Introduction

SlapOS Introduction

This visual guide has been created to introduce SlapOS to developers and users interested in understanding or marketing SlapOS. It will explain what SlapOS is, show how to get started with SlapOS and describe the underlying architecture.

Table of Content

  • What is SlapOS?
  • Demo
  • SlapOS Architecture
  • Using SlapOS

What is SlapOS?

This section explains the basics about SlapOS, its philosophy and in what situations SlapOS can be a useful solution.

SlapOS

SlapOS Graphic - Overview

SlapOS stands for Simple Laguage for Accounting and Provisioning Operating System. It is a general purpose operating system usable on distributed POSIX infrastructures (Linux, xBSD) to provide and manage software services without the need for virtualisation. SlapOS is published as Free Software under fairly permissive licensing.

SlapOS is useful for any company which needs to run more than 20 instances of the same application or which needs to operate more than 10 servers. SlapOS is a possible solution for a variety of scenarios such as:

  • public or private cloud computing infrastructure (IaaS, PaaS, SaaS, DraaS)
  • hyperconverged infrastructure (virtualisation of compute, storage and network)
  • edge computing infrastructure (CDN, IoT gateway, AI offloading, cache, buffering)
  • 4G/5G network management system (eNodeB, locale core network)
  • resilient low latency network as a service (IPv4, IPv6, RINA)
  • tactical cloud bubble

Service and Process Oriented System

SlapOS Graphic - Network

On a simplified level, SlapOS operates a network of machines. A Master (COMP-ROOT) contains a catalog of software services which are published on Slave nodes (COMP-0,1,2,3) with the master tracking service consumption and providing the means for managing and invoicing of specific services.

SlapOS can be described as a cloud operating system in which "everything is a process" unlike Unix in which "everything is a file". If one has to manage thousands of servers with thousands of processes, hundred different applications in multiple different releases or versions, SlapOS is useful by making the whole management process well specified, automated and under controllable.

By writing a simple recipe using a software called Buildout, SlapOS can help you turn any software application (open source or proprietary) into a solid SaaS or PaaS platform. SlapOS handles provisionning of application instances, accounting billing for instance usage. SlapOS provides a Dashboard and interfaces to handle customer registration, support request management, invoicig and payment.

Philosophy

SlapOS Graphic - Philosophy

The underlying philosophy of SlapOS was created by ViFiB, one of the first providers of "Distributed Cloud Computing".

The first idea in SlapOS is to consider that "everything is a process" and that a system is made of processes which exchange information through network services based on various protocols (HTTP, memcached, mysql, et al).

The second idea in SlapOS is that the best way to create a reliable and efficient cloud computing system is to follow the original principles of the Internet: distribution and simplicity. Rather than trying to build one or two reliable data centers with redundant network, redundant power supply, redundant air conditionning, redundant staff, etc. in order to reach 99.999% availability, SlapOS aggregates unreliable hosting, such as home cloud or low cost dedicated servers and, by combining them, reaches a reliability of 99.9999%. SlapOS is in reality much more reliable than traditional approaches to Cloud operation because, by selecting independent sources, SlapOS can mitigate the risk of major system outages due to electricity failures, earthquakes, political decisions, etc. That said, SlapOS is and can also be used to operate traditional data centers more efficiently.

The third idea in SlapOS is that any intelligent process to ensure redundancy, data replication or energy savings should be implemented by the application itself rather than by the infrastructure. This is in line with the original ideas behind the Internet: a "stupid" network running clever applications. By taking into account usage or data specificities, optimal policies for redundancy and energy savings can be implemented.

Demo

The following section contains screenshots of how to request a software service from ViFiB, creator of the SLAP protocol and SlapGRID. ViFiB leads the SlapOS project which is based on scientific ideas from Prof. Christophe CĂ©rin of Paris University (XIII).

Besides developing custom implementations for clients, ViFiB also provides a reference implementation of SlapOS which is used to host Software as a Service (SaaS), Virtual Machines and a federated Infrastructure as a Service (IaaS) to Nexedi, the mother company of ViFiB. This SlapOS implementation is used to provide various software services to developers in the Nexedi offices around the world.

For the demo a user account (commercial service) on Vifib is required.

Vifib Dashboard

Vifib Interface - Dashboard

After logging into the SlapOS Dashboard, a user can request new software services using the Services menu in the left hand panel.

Vifib Instantiated Services

Vifib Interface - Instantiated Services

The Services section displays the list of currently instantiated services. These can be complete business applications (eg ERP5, Drupal, XWiki), well defined network services (eg. memcached, cloudooo, Frontend) which may be part of a Service Orientated Architecture (SOA), a virtual machine (eg KVM) or an instance of another IAAS provider (eg. Amazon EC2).

To instantiate a new service, click Add in the subheader.

Vifib Available Services

Vifib Interface - Available Services

This will load the list of available services on Vifib. Select the service to be installed.

Vifib Available Service Versions

Vifib Interface - Available Service Versions

Once a service is selected, a version must also be selected. SlapOS is capable of providing multiple instances of the same application in multiple versions or releases (eg. users may wish to use MySQL 5.0 or MySQL 5.1, MariaDB or SkySQL).

There is no limitation in SlapOS in terms of number of versions of releases available for each application. This is very important for real production systems which often require to patch files or change compilation options of the underlying base software. Here SlapOS departs from traditional GNU/LInux distributions in the sense that it does not try to define what is a "standard MySQL" or "standard Java" or even a "standard Apache". Experience showed that minor, yet essential changes to any software are eventually required to build a scalable system on the Cloud. GNU/Linux distributions cannot be trusted beyond Linux kernel and core libraries for that purpose.

It was found for example that some GNU/Linux distributions add bugs to packaged software in the process of making it compliant with the distribution guidelines. Bugs can sometimes lead to unit tests not passing anymore. Messages in the distribution bug tracker sometimes state that there is no plan to fix this bug since there is no way to solve in a way consistent with the distribution guidelines. This can make sense from the GNU/Linux distribution point of view but may be unacceptable to build a production system.

Configure Instance

Vifib Interface - Service Configuration

Before the service is requested, the a number of software-specific configuration parameters can be provided. Manual selection of a network node will override the automatic allocation of the service by the SlapOS Master.

The request is triggered by clicking Proceed. Once the software has been installed in a computer partition of the chosen node and is ready to be used, its connection parameters will be published on a new entry in the Server List previously seen. (for more detailed information and documentation, please refer to the SlapOS Learning Track).

Note, that while the Vifib implementation of SlapOS does not use invoicing for software instances or consumption as they are only used internally, these features are trivial to integate if required.

SlapOS Architecture

The following section briefly describes the architecture underlying architecture of SlapOS. A more detailed presentation is available at understanding the SlapOS architecture.

Master and Nodes

SlapOS Master and Nodes

The SlapOS architecture uses two types of servers: SlapOS Master (COMP-ROOT) and SlapOS Nodes (COMP-0,1,2,3).

SlapOS Nodes are hosted in multiple data centers or at home. They exchange information with the SlapOS Master through the SLAP protocol. SLAP stands for "Simple Language for Accounting and Provisioning".

The SlapOS Master tells to each SlapOS Node which software it should install and which instances of which software should be run. Each SlapOS Node tells the SlapOS Master how much resources were consumed by each instance of each software.

Based on the start dates and stop dates which represent the time series of execution frames of each application, and based on the resource usage of each application, SlapOS master can produce invoices for every user of SlapOS based on his personal usage.

The current architecture of SlapOS, based on a single redundant master node, is designed to manage thousands of servers. Future versions of SlapOS may however rely on a distributed implementation SlapOS Master, too.

SlapOS Node Initial State

SlapOS Node in Initial State

SlapOS Nodes run a small kernel called the "SlapOS Kernel". This kernel consists of a minimal GNU/Linux distribution, the SLAPGrid daemon, a buildout environment for bootstrapping applications and supervisord to control running processes.

The current implementation of "SlapOS Kernel" was made using OpenSuSE and the SuSE Studio service. "SlapOS Kernel" is small enough to fit on an USB key.

SlapOS Master Node

SlapOS Master Node

The SlapOS Master is simply a standard SlapOS Node with ERP5 installed.

This ERP5 "Cloud Engine" is an extension of ERP5 open source ERP which provides, in addition to invoicing and financials, all the standard business processes required to operate an IaaS, SaaS or PaaS company. ERP5 Cloud Engine can be used standalone or in combination with other business applications.

The core scope of ERP5 Cloud Engine is to keep track of all available SlapOS Nodes, to list all software applications and releases which can be requested and define how they can be installed, to define which releases of which applications may be installed on which SlapOS nodes, to specify how a given application can be instantiated and to keep track of which instances are being run on which SlapOS Nodes. The core scope of ERP5 Cloud Engine is also called sometimes "SlapGRID Cloud Engine".

The ERP5 Cloud Engine implements in addition the subscription process for users to register to SaaS, PaaS or IaaS, an e-commerce portal for users to order and access their instances through the Web, a CRM for support staff to handle support requests and support documentation, and a complete financial system.

ERP5 Cloud Engine can be interfaced to existing systems using standard interfaces based on standard protocols such as SyncML, REST, SOAP, XML-RPC, LDAP or MySQL. ERP5 Cloud Engine supports most SSO implementations too.

SlapOS Node Installed

SlapOS Node in Installed State

After a SlapOS Node is initially installed, a user may request an instance of a given (and available) software to be installed. This is achieved via so-called "recipes" and "software releases".

SlapOS already provides its own default recipes for many common softwares (MySQL, Apache, memcached, et al.). However, it is recommended for users to always create their own recipes with their own compilation options and specific build configuration. As an example, a recipe can created for specific version of MediaWiki and another one for OS Commerce. Each recipe may embed is own version of Apache, MySQL, etc. with its own custom compilation options. This way, a recipe is really self-contained and completely under control of the recipe author.

A Software Release is defined as the URL and revision number of a Subversion (SVN) repository which contains a collection of recipes required to build a complete software application. This repository is called "buildout directory". Software releases also define a signature in order to make sure that buildout directories which are downloaded contain the same files as those which are expected.

The process of installing the Software Release relies on two components of the SlapOS Kernel: SLAPgrid and buildout. SLAPgrid requests to SlapOS Master the list of software which should be installed on the SlapOS Node. SlapOS Master returns a list of Software Releases to install. SLAPgrid downloads through SVN the buildout directory of the Software Release and runs the buildout command. The buildout command downloads all source code and dependencies required to compile the software application defined in the recipes, compiles all source code (using cache to accelerate compilation) and installs software application binaries, files and libraries in a subdirectory of /opt.

SlapOS Node Running

SlapOS Node in Running State

Once the requested applications are installed on a SlapOS node using the concept of "Software Release", it is possible to create so-called "Software Instances".

The concept of software instance refers to the idea that modern servers can easily run multiple independent processes of the same software. With IPv6, it is also quite easy to associate to every server hundreds of independent IP adresses, one for each instance of each application. And because all applications share the same shared libraries, memory footprint overhead to run yet another instance of the same software is minimal, unlike what happens with virtual machines.

It is thus possible to run on a single SlapOS more than 200 instances of the same software. In the case of the TioLive service (discontinued), 200 ERP5 open source ERP instances were run on a standard dual CPU server.

Using SlapOS

The following section describes how to use SlapOS from installation to creating recipes and points to the relevant documentation.

SlapOS Slap Installation

SlapOS Icon - Command Line Installer

Using SlapOS is really easy. It requires to install Re6st (if IPv6 is not available by default) to connect with a SlapOS network and then installing a server, which can be done in various ways, most easily using the recommended single line installer. It only takes a few minutes.

Setting up a SlapOS Master requires more steps described in installing and configuring a SlapOS Master.

Simple Slap Recipes

class Recipe(BaseSlapRecipe):
  def_createConfigurationString(self):
    instance_parameterdict = self.computer_partition\
      getInstanceParameterDictu()
    ip = isntance_parameter_dict['ip_list'][0][1]
    tcp_port = 51000
    self.computer_partition.setConnectionDict(
      dict(ip=ip,tcp_port=tcp_port
    )
    return '-|%s -p %s -U 0' (ip, tcp_port)

  def_install(self):
    self.wrapper_template_location = pkg_resources.resource_filename(
      __name__, os.path.join('template','wrapper.in')
    )
    self._createRunningWrapper(dict(
      real_binary = self.options['server_binary'],
      configuration_string = self.createConfigurationString()
    ))
    # nothing to return to buildout
    return []

Creating recipes for SlapOS is also quite straightforward. In the above example, a memcached server is instanciated. In order to start this service, it is simpy required to generate a startup script, which uses the connection informations returned by the SlapOS Master.

SlapOS Requesting

> python
from slapos.slap import slap
import memcache

s = slap()
s.initializeConnection(
  "https://nexedivifib1.dyn.majimoto.net:40443/erp5/portal_salp"
)
cp = s.registerComputerPartition("nexedivifib8", "slappart197")
ncp = cp.request(
  "https://nexedivifib1.dyn.majimoto.net:40443/erp5/web_site_module/erpypi/
  memcached_software_profile-0.0.3.cfg", "memcached", "memcached_server 1")
...
connection = ncp.getConnectionDict()
mc = memcache.Client(['%s:%s' % (
  connection['ip'], connection['tcp_port'])])
mc.set("some_key", "Some Value")
value = mc.get("some_key")

The SLAP protocol itself if very easy to invoke using the SLAP library. In the above example a new connection to the SlapOS Master is initialized and an existing computer partition object is instantiated. From this one, a new service is requested (in this case, a memcached server).

After waiting a few seconds until this new service has been instantiated, and by using the standard memcache library, the memcached server can be accessed and used.

XXX Slap Console example XXX

SlapOS Compatabiilty

SlapOS Custom Application SlapOS Stack

SlapOS has been designed for maximum compatibility. SlapOS default distribution already contains recipes for many NoSQL technologies, databases, runtimes and application servers. Recipes that do not exist yet are either planned or easy to derive from the thousands of existing community created buildout recipes.

Making your own contribution is thus just as easy as copying ideas from others and adapting them to your needs. For example, creating a high performance LAMP recipe for SlapOS consists of reusing a couple of existing recipes for Apache, MySQL, Varnish, ha-proxy and combining them so that the PHP application can be distributed across multiple servers sharing two MySQL databases configured in redundant mode. Then increasing from 1 to 2 to 3 and then to 100 the number of Apache nodes which are instanciated on the SlapOS cloud is just a matter of changing a parameter of the recipe. SlapOS takes care of the rest.

Thank You

Image Nexedi Office
  • Nexedi GmbH
  • 147 Rue du Ballon
  • 59110 La Madeleine
  • France