Extend Software Release

Extend Software Release

Extend Software Release

This document will walk through the steps of extending an existing software release available on SlapOS. It will demonstrate how to add parameters, components and features.

This document does not cover creating a software release itself.

To follow the steps in this document, you should have read understanding SlapOS architecture and Buildout. It is also required to have:

Table of Content

  • Add Parameter to all Software Instances
  • Add Feature
  • Add Component

Add Parameter to all Software Instances

This section will modify the "helloworld" Software Release, a software that creates a simple web server and publishes the text hello <configuration.name>.

This will be done using a Webrunner a development IDE, that contains a "mini SlapOS Master" for deploying a single software (helloworld). In more complex SlapOS networks, a master manages a catalog of softwares which are supplied and instantiated on a network of nodes, some of which may again be Webrunners containing a single software release. Webrunners are useful, because they provide resiliency for this single software by hosting multiple "internal" instances. The deployed software may also be patched and can be accessed via SSH which for some softwares may be required to access the underlying file system.

The software release of "Helloworld" is relatively simple and composed of two files:

  • software.cfg - the Buildout profile which installs the software release on a computer
  • instance.cfg.in - the instance profile which will be filled by software.cfg

instance.cfg.in will be used to deploy (configure and run) one (Webrunner) or multiple instances (SlapOS network) of a software. Note, that after Buildout has filled the instance.cfg.in file, there should be a hidden file called .installed.cfg. In case of instantiation failing this is a good place to look for what went wrong (see also the chapter on debugging the Webrunner in debugging a SlapOS node).

Check Connection Parameters

Webrunner Interface - Extending Software Release - Verify Connection Information

Log in to the webrunner and open the Services tab. Check the Connection Information Slap Response to see which parameters the helloworld software release is currently outputting. The connection urls of the webservers in Go, Ruby and Python are also outputted. Opening either one will show the default page with the name parameter.

Locate Helloworld Software.cfg

Extending Software Release - Webrunner Interface - Fullscreen Editor

Head over to the Editor. Click the fullscreen icon in the toolbar, then locate the Helloworld folder in the Software directory (see understanding SlapOS basics for more information on the folder structure of the SlapOS node).

Software.cfg

Extending Software Release - Webrunner Interface - Helloworld Software Config

All software.cfg profiles extend the default slapos.cfg (which contains basic slapos deployment tools). In this example it is extended by ../../component/helloweb/buildout.cfg - the profile to create some programs that say "hello" in three different languages (python, ruby, golang). It also describes how to parse the instance profile instance.cfg.in into the rendered file later used for supplying instances of our helloworld software.

Note the comment regarding the MD5 checksum and then either comment out your checksum or replace it with the hash of [your_instance.cfg.in]. Otherwise the build will fail when checking the integrity of the file.

One way to customize a software instance is through providing instance parameters when requesting an instance from a SlapOS Master using the instance XML configuration. Let's add a new parameter called title to display the title of a web page alongside the name. Open the instance.cfg.in in the same directory.

Instance.cfg.in

Extending Software Release - Webrunner Interface - Helloworld Instance Config Template

In the instance.cfg.in [instance-parameter] section, add a new parameter like so:

# XXX This is the declaration of the default value for the parameter "title" 
# (None here). If commented, the recipe will crash. The "configuration." part 
# of the parameter name means that the value will be the one given (if present) 
# in instance parameters
configuration.title =

And in the [publish-connection-parameter] section, output the new parameter as well:

# XXX the published parameter "title". By default, its value is the value 
# declared in the section [instance-parameter] above.
title = Title ${instance-parameter:configuration.title}!

Rebuild Software Release

Extending Software Release - Webrunner Interface - Software Rebuild

Save the modified file in the Webrunner, don't forget to uncomment the MD5 checksum in the software.cfg, then hit the green play button in the top right toolbar to rebuild your Webrunner (you may have to get out of fullscreen mode to show the green button again). Rebuilding should take a few minutes and will not complete in case you have made an error when updating the files.

Successful Rebuild

Extending Software Release - Webrunner Interface - Software Rebuild Completed

Once both indicators switch to completed and green status the helloworld software has been rebuilt with the modificed configuration files. Head back to Services to check the Connection Information.

Updated Software Connection Information

Extending Software Release - Webrunner Interface - Software Rebuild Completed

The title parameter is now displayed as well. As we did not provide a parameter akin to John Doe for the name, nothing is displayed. XXX HOW TO ADD PARAMETER TO DASHBOARD? XXX

While this update was done in a Webrunner and therefore only on a single instance, it can just as well be done on a SlapOS network. In this case, the Software Release points to a repository with the software.cfg and instance.cfg.in. For more details on the equivalent steps to rebuilding the software, please look at how to update a software release.

Add Feature

After working with existing sections, the following section will show how to add new sections using different recipes. We will be adding a template file and render it during instantiation.

Add Template Folder and Template

Extending Software Release - Webrunner Interface - Add Template Folder and Template

Head back to the Editor and create a new folder named /template. Create a new file named rendered.in and open it.

SlapOS software configuration templates are using Jinja2 (more info on the jinja2 structure and syntax). Add the following content to the file and save:

{% if title %}
{{ title }}
-
{% endif %}
Title: {{ name }}!

In order to import this template into the software release, the software profile needs to be modified.

Add Template to Software Profile

Extending Software Release - Webrunner Interface - Add Template to Software Profile

Note that there are now different templating formats being used:

  • Buildout templates using object['parameter'] and ${:variable name}
  • Jinja templates using the syntax: {{ }}

First, in software.cfg add a new entry to the parts section called template-get (Note, that get is an arbitrarly chosen name given to our template). Then update the [instance-profile] context (see jinja specifications on context) to include a second section to be available in instance.cfg.in:

context =
  section buildout buildout
  section template_get template-get

Note for exposing context parameters, there must always be three parts (type, name and expression) separated by whitespace, that multiple entries go on separate lines and that the name cannot contain "-", because this will be the caller in later requests.

Finally add the corresponding section to the end of the file:

# Retrieve Jinja2 template to render via url. Location is only used 
# to share information through buildout (corresponds to a default location)
[template-get]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/template/${:filename}
location = ${buildout:parts-directory}/${:_buildout_section_name_}
filename = rendered.in
download-only = true
extensions = jinja2.ext.do
context =
  section buildout buildout

Note that this defines the location and filename variables we will require in the next step. Head over to instance.cfg.in

Update Instance.cfg.in

Extending Software Release - Webrunner Interface - Add Template to Instance Profile

As in the software profile, start by adding a new entry called template-creation to the [buildout] parts section along with a new folder after the var and etc folders using:

# Create folder which will contain the jinja template
data = ${:home}/srv/data

Then add a new section:

# XXX Jinja action!
[template-creation]
recipe = slapos.recipe.template:jinja2

# The template value will be replaced via the info given in software.cfg
template = {{ template_get['location'] }}/{{ template_get['filename'] }}

# Declare where we want to get the new file rendered by Jinja
rendered = ${directory:data}/data1.txt
mode = 700

# Using context we can define specific values wich will be substituted in the 
# template. "key" means that the value is defined by buildout.
context = 
  key name instance-parameter:configuration.name
  key title instance-parameter:configuration.title 

Note, the use of template_get which we defined earlier in the software.cfg context of the [instance-profile]

Also, it is possible define the template inline.

# template = inline:
#  ----------------------------------
#  Title : {{title}}
#  ----------------------------------
#  Hello {{name}}

Save the file and rebuild by pressing the green play button again.

Verify Template is Available

Extending Software Release - Webrunner Interface - Verify Template is Available

If everything was done correctly you should find the rendered template at ${template-creation:rendered}, which should compute to the the srv/data/data1.txt on your instance.

There are plenty of pre-existing recipes available online (see the buildout documentation) as well as in the Slapos Git repository.

Should you run into issues, make sure to check the software and instance logs in the Logs section and make sure that sections you have defined (like template-get and template-creation) show up in the logs and can be processed without errors.

Add Component

It is common having to add a new component to a software release in order to add certain features. There is a number of components available to use from the SlapOS repository.

This section will implement a log rotation feature using logrotate, cron and gzip.

Add Logrotate To Software.cfg

Extending Software Release - Webrunner Interface - Add Logrotate to Software Profile

Start by adding the required components to the extends section:

# Add needed components
  ../../component/dcron/buildout.cfg
  ../../component/logrotate/buildout.cfg
  ../../component/gzip/buildout.cfg

and in the parts section, add the deployment entries:

# Deployment of new components.
  dcron
  logrotate

Logrotate allows rotation of log files and cron will execute them regularly. Gzip is just added to have a more concrete example. Next update they instance.cfg.in.

Configure Components in instance.cfg.in

Extending Software Release - Webrunner Interface - Add Logrotate to Instance Profile

Start by creating directories for configs and logs. Comment out the current data folder created for the jinja template earlier, then add the following folders to your [directory] recipe:

bin = ${:home}/bin
srv = ${:home}/srv
data = ${:srv}/data
# path of the log directory used by our service (see [hello-world])
log = ${:var}/log

# cron directories
cronentries = ${:etc}/cron.d
crontabs = ${:etc}/crontabs
cronstamps = ${:etc}/cronstamps

# logrotate directories
backup = $${:srv}/backup
logrotate-backup = $${:backup}/logrotate
logrotate-entries = $${:etc}/logrotate.d

Next we'll install and deploy cron

Add a new section for cron:

# Deploy cron service
[cron]
recipe = slapos.cookbook:cron
dcrond-binary = ${dcron:location}/sbin/crond
cronentries = ${directory:cronentries}
crontabs = ${directory:crontabs}
cronstamps = ${directory:cronstamps}
catcher = ${cron-simplelogger:wrapper}
binary = ${directory:service}/crond

# Add logger to cron
[cron-simplelogger]
recipe = slapos.cookbook:simplelogger
wrapper = ${directory:bin}/cron_simplelogger
log = ${directory:log}/cron.log

# Add a cron entry
[cron-entry-logrotate]
<= cron
recipe = slapos.cookbook:cron.d
name = logrotate
frequency = 0 * * * *
# Now that the cron is set up, we can define
# the command to launch at the defined frequency
command = $${logrotate:wrapper}

First, we deploy cron using cron's recipe. In this recipe we use as a catcher for logs with a wrapper defined in the cron-simplelogger section. These buildout sections where copied from the apache-frontend software profile. It means that it easy to reuse sections from other profiles if they provide features matching your needs.

Now that the cron is set up, we only need to add the command to launch logrotate at the defined frequency:

command = $${logrotate:wrapper}
The last step is adding the logrotate itself:
# Deploy Logrotate (see ../../slapos/recipe/logrogate.py)
[logrotate]
recipe = slapos.cookbook:logrotate

# Binaries
logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
gzip-binary = ${gzip:location}/bin/gzip
gunzip-binary = ${gzip:location}/bin/gunzip

# Directories
wrapper = $${directory:bin}/logrotate
conf = $${directory:etc}/logrotate.conf

# in this folder will be defined tho files to save, and how to save them
logrotate-entries = $${directory:logrotate-entries}

# in this folder the logs will be compressed then saved
backup = $${directory:logrotate-backup}
state-file = $${directory:srv}/logrotate.status

[logrotate-entry]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = helloworld-logrotate
log = $${directory:log}/log.log
frequency = daily
rotatep-num = 30
sharedscripts = true
notifempty = true
create = true

Rebuild Software

Extending Software Release - Webrunner Interface - Rebuild Software with Logrotate

Once all changes have been made, save the files and rebuild the software clicking the green button. It should finish without problems.

Verify Logs Are Running

Extending Software Release - Webrunner Interface - Verifiy Logs are Avaiable

Once the rebuild finished, head to Editor and check whether the instance includes the required log files.

Update MD5 Sum

Extending Software Release - Webrunner Interface - Update MD5 Sum

Once you finished all modifications, don't forget to uncomment and update the MD5 Sum. On your active file, click the Menu button in the subheader and select Get or Update MD5 sum. The MD5 sum will be displayed on the bottom of the screen. Paste it into your software.cfg profile and rebuild again.

Thank You

Image Nexedi Office
  • Nexedi SA
  • 147 Rue du Ballon
  • 59110 La Madeleine
  • France