This document will walk through the steps of extending an existing software release available on SlapOS. It will demonstrate how to add parameters, components and features.
To follow the steps in this document, you should have read understanding SlapOS architecture and be familiar with Buildout. It is also required to have the Webrunner Software added to the SlapOS Master software catalouge and have an instance of the Webrunner with the included helloworld software release deployed inside.
This section will modify the "helloworld" Software Release, a software that creates a simple web server and publishes the text hello <configuration.name>.
This will be done using a Webrunner a development IDE, that contains a "mini SlapOS Master" for deploying a single software (helloworld). In more complex SlapOS networks, a master manages a catalog of softwares which are supplied and instantiated on a network of nodes, some of which may again be Webrunners containing a single software release. Webrunners are useful, because they provide resiliency for this single software by hosting multiple "internal" instances. The deployed software may also be patched and can be accessed via SSH which for some softwares may be required to access the underlying file system.
The software release of "Helloworld" is relatively simple and composed of two files:
software.cfg
- the Buildout profile which installs the software release on a computerinstance.cfg.in
- the instance profile filled by software.cfg
on instantiationinstance.cfg.in
will be used to deploy (configure and run) one (on the Webrunner) or multiple (on a SlapOS network) instances of a software. Note, that after Buildout has filled the instance.cfg.in
file, there should be a hidden file called .installed.cfg
. In case of instantiation failing this is a good place to look for what went wrong.
Log in to the webrunner and open the Services tab. Check the Connection Information Slap Response to see which parameters the helloworld software release is currently outputting. The connection urls of the webservers in Go, Ruby and Python are also outputted. Opening either one will show the default page with the name parameter.
Head over to the Editor. Click the fullscreen icon in the toolbar, then locate the Helloworld folder in the Software directory.
All software.cfg
profiles extend the default slapos.cfg (which contains basic slapos deployment tools). In this example it is extended by ../../component/helloweb/buildout.cfg
- the profile to create some programs that say "hello" in three different languages (python, ruby, golang). It also describes how to parse the instance profile instance.cfg.in
into the rendered file later used for supplying instances of our helloworld software.
Note the comment regarding the MD5 checksum. MD5 checksum is stored in buildout.hash.cfg,
either comment out your checksum or replace it with the hash of [your_instance.cfg.in]
. Otherwise the build will fail when checking the integrity of the file.
One way to customize a software instance is through providing instance parameters when requesting an instance from a SlapOS Master using the instance XML configuration. Let's add a new parameter called title to display the title of a web page alongside the name. Open the instance.cfg.in
in the same directory.
In the instance.cfg.in
[instance-parameter]
section, add a new parameter like so:
# XXX This is the declaration of the default value for the parameter "title"
# (None here). If commented, the recipe will crash. The "configuration." part
# of the parameter name means that the value will be the one given (if present)
# in instance parameters
configuration.title =
And in the [publish-connection-parameter]
section, output the new parameter as well:
# XXX the published parameter "title". By default, its value is the value
# declared in the section [instance-parameter] above.
title = Title ${instance-parameter:configuration.title}!
Save the modified file in the Webrunner then hit the green play button in the top right toolbar to rebuild your Webrunner (you may have to get out of fullscreen mode to show the green button again). Rebuilding should take a few minutes and will not complete in case you have made an error when updating the files.
Once both indicators switch to completed and green status the helloworld software has been rebuilt with the modified configuration files. Head back to Services to check the Connection Information.
The title
parameter is now displayed as well. As we did not provide a parameter akin to John Doe for the name
, nothing is displayed.
While this update was done in a Webrunner and therefore only on a single instance, it can just as well be done on a SlapOS network. In this case, the Software Release points to a repository with the software.cfg
and instance.cfg.in
. Updating these profiles and then adding a new version entry to the SlapOS Master software catalogue will trigger a pending upgrade of all instances of this software.
After working with existing sections, the following section will show how to add new sections using different recipes. We will be adding a template file and render it during instantiation.
Head back to the Editor and create a new folder named /template
. Create a new file named rendered.in
and open it.
SlapOS software configuration templates are using Jinja2 (more info on the jinja2 structure and syntax). Add the following content to the file and save:
{% if title %}
{{ title }}
-
{% endif %}
Title: {{ name }}!
In order to import this template into the software release, the software profile needs to be modified.
Note that there are now different templating formats being used:
object['parameter']
and ${:variable name}
{{ }}
First, in software.cfg
add a new entry to the parts
section called template-get (Note, that get is an arbitrarly chosen name given to our template). Then update the [instance-profile]
context (see jinja specifications on context) to include a second section to be available in instance.cfg.in
:
context =
section buildout buildout
section template_get template-get
Note for exposing context parameters, there must always be three parts (type, name and expression) separated by whitespace, that multiple entries go on separate lines and that the name cannot contain "-", because this will be the caller in later requests.
Finally add the corresponding section to the end of the file:
# Retrieve Jinja2 template to render via url. Location is only used
# to share information through buildout (corresponds to a default location)
[template-get]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/template/${:filename}
location = ${buildout:parts-directory}/${:_buildout_section_name_}
filename = rendered.in
download-only = true
extensions = jinja2.ext.do
context =
section buildout buildout
Note that this defines the location and filename variables we will require in the next step. Head over to instance.cfg.in
As in the software profile, start by adding a new entry called template-creation to the [buildout] parts
section along with a new folder after the var and etc folders using:
# Create folder which will contain the jinja template
data = ${:home}/srv/data
Then add a new section:
# XXX Jinja action!
[template-creation]
recipe = slapos.recipe.template:jinja2
# The template value will be replaced via the info given in software.cfg
template = {{ template_get['location'] }}/{{ template_get['filename'] }}
# Declare where we want to get the new file rendered by Jinja
rendered = ${directory:data}/data1.txt
mode = 700
# Using context we can define specific values wich will be substituted in the
# template. "key" means that the value is defined by buildout.
context =
key name instance-parameter:configuration.name
key title instance-parameter:configuration.title
Note, the use of template_get which we defined earlier in the software.cfg
context of the [instance-profile]
Also, it is possible define the template inline.
# template = inline:
# ----------------------------------
# Title : {{title}}
# ----------------------------------
# Hello {{name}}
Save the file and rebuild by pressing the green play button again.
If everything was done correctly you should find the rendered template at ${template-creation:rendered}
, which should compute to the the srv/data/data1.txt
on your instance.
There are plenty of pre-existing recipes available online (see the buildout documentation) as well as in the Slapos Git repository.
Should you run into issues, make sure to check the software and instance logs in the Logs section and make sure that sections you have defined (like template-get and template-creation) show up in the logs and can be processed without errors.
It is common having to add a new component to a software release in order to add certain features. There is a number of components available to use from the SlapOS repository.
This section will implement a log rotation feature using logrotate, cron and gzip.
Start by adding the required components to the extends
section:
# Add needed components
../../component/dcron/buildout.cfg
../../component/logrotate/buildout.cfg
../../component/gzip/buildout.cfg
and in the parts
section, add the deployment entries:
# Deployment of new components.
dcron
logrotate
Logrotate allows rotation of log files and cron will execute them regularly. Gzip is just added to have a more concrete example. Next update the instance.cfg.in
.
Start by creating directories for configs and logs. Comment out the current data folder created for the jinja template earlier, then add the following folders to your [directory]
recipe:
bin = ${:home}/bin
srv = ${:home}/srv
data = ${:srv}/data
# path of the log directory used by our service (see [hello-world])
log = ${:var}/log
# cron directories
cronentries = ${:etc}/cron.d
crontabs = ${:etc}/crontabs
cronstamps = ${:etc}/cronstamps
# logrotate directories
backup = ${:srv}/backup
logrotate-backup = ${:backup}/logrotate
logrotate-entries = ${:etc}/logrotate.d
Next we'll install and deploy cron
Add a new section for cron:
# Deploy cron service
[cron]
recipe = slapos.cookbook:cron
dcrond-binary = ${dcron:location}/sbin/crond
cronentries = ${directory:cronentries}
crontabs = ${directory:crontabs}
cronstamps = ${directory:cronstamps}
catcher = ${cron-simplelogger:wrapper}
binary = ${directory:service}/crond
# Add logger to cron
[cron-simplelogger]
recipe = slapos.cookbook:simplelogger
wrapper = ${directory:bin}/cron_simplelogger
log = ${directory:log}/cron.log
# Add a cron entry
[cron-entry-logrotate]
<= cron
recipe = slapos.cookbook:cron.d
name = logrotate
frequency = 0 * * * *
# Now that the cron is set up, we can define
# the command to launch at the defined frequency
command = ${logrotate:wrapper}
First, we deploy cron using cron's recipe. In this recipe we use as a catcher for logs with a wrapper defined in the cron-simplelogger section. These buildout sections where copied from the apache-frontend software profile. It means that it easy to reuse sections from other profiles if they provide features matching your needs.
Now that the cron is set up, we only need to add the command to launch logrotate at the defined frequency:
command = ${logrotate:wrapper}
The last step is adding the logrotate
itself:
# Deploy Logrotate (see ../../slapos/recipe/logrogate.py)
[logrotate]
recipe = slapos.cookbook:logrotate
# Binaries
logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
gzip-binary = ${gzip:location}/bin/gzip
gunzip-binary = ${gzip:location}/bin/gunzip
# Directories
wrapper = ${directory:bin}/logrotate
conf = ${directory:etc}/logrotate.conf
# in this folder will be defined tho files to save, and how to save them
logrotate-entries = ${directory:logrotate-entries}
# in this folder the logs will be compressed then saved
backup = ${directory:logrotate-backup}
state-file = ${directory:srv}/logrotate.status
[logrotate-entry]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = helloworld-logrotate
log = ${directory:log}/log.log
frequency = daily
rotatep-num = 30
sharedscripts = true
notifempty = true
create = true
Once all changes have been made, save the files and rebuild the software clicking the green button. It should finish without problems.
Once the rebuild finished, head to Editor and check whether the instance includes the required log files.
Once you finished all modifications, don't forget to uncomment and update the MD5 Sum. On your active file, click the Menu button in the subheader and select Get or Update MD5 sum. The MD5 sum will be displayed on the bottom of the screen. Paste it into your buildout.hash.cfg profile and rebuild again.
Promise is an executable doing some arbitrary work, then exiting with exit code 0 ("it works") or greater ("it doesn't work"). They are run by slapgrid to know whether an instance is working or not.
This section will add a simple promise to the helloworld software profile.
Writing a promise consists of defining a class called RunPromise which inherits from the GenericPromise class and defines three methods:
from zope.interface import implementer
from slapos.grid.promise import interface
from slapos.grid.promise.generic import GenericPromise
import socket
import sys
@implementer(interface.IPromise)
class RunPromise(GenericPromise):
def __init__(self, config):
super(RunPromise, self).__init__(config)
# check port is listening at least every 2 minutes
self.setPeriodicity(minute=2)
def sense(self):
"""
Simply test if we can connect to specified host:port.
"""
hostname = self.getConfig('hostname')
# if type of port should be int or str, unicode is not accepted.
port = int(self.getConfig('port'))
addr = (hostname , port)
# in case of any error, we call "logger.error"
# note that we could simply let the function raise an error, it would have the same effect
# in this case the following code would be enough:
# socket.create_connection(addr).close()
# self.logger.info("port connection OK")
try:
socket.create_connection(addr).close()
except (socket.herror, socket.gaierror) as e:
self.logger.error("ERROR hostname/port ({}) is not correct: {}".format(addr, e))
except (socket.error, socket.timeout) as e:
self.logger.error("ERROR while connecting to {}: {}".format(addr, e))
else:
self.logger.info("port connection OK ({})".format(addr))
# no need to define Test as it is the default implementation
#def test(self):
# """
# Test is failing if last sense was bad.
# """
# return self._test(result_count=1, failure_amount=1)
def anomaly(self):
"""
There is an anomaly if last 3 senses were bad.
"""
return self._anomaly(result_count=3, failure_amount=3)
The most simple example of promise is "check_if_port_listening" (https://lab.nexedi.com/nexedi/slapos.toolbox/blob/master/slapos/promise/plugin/check_port_listening.py) trying to open a socket to an ip/port. If it works, the promise exits with exit code 0, and slapgrid knows that the instance is working. If the socket can't be created, it exits with a different exit code, and slapgrid reports it to the SlapOS Master.
For convenience, we will get use of [monitor-promise-base]
already defined at https://lab.nexedi.com/nexedi/slapos/blob/master/stack/monitor/instance-monitor.cfg.jinja2.in:
[monitor-promise-base]
recipe = slapos.cookbook:promise.plugin
eggs =
slapos.toolbox
content =
from slapos.promise.plugin.${:module} import RunPromise
mode = 600
output = ${directory:plugins}/${:name}
Because of extends = {{ template_monitor }}
present in your instance profile (check that it is there) we can utilize it really easily. In your instance profile, add the following code (feel free to check any other site instead of erp.com):
# url checking promise
[url-promise]
<= monitor-promise-base
module = check_url_available
name = check_some_url.py
config-url = https://erp5.com
config-check-secure = 1
Note <=
syntax being used to extend a section already defined elsewhere, in our case [monitor-promise-base] shown above.
Don't forget to add url-promise to the parts
section.
This will fetch the promise check_url_available.py
from https://lab.nexedi.com/nexedi/slapos.toolbox/tree/master/slapos/promise/plugin and generate an executable ${directory:plugins}/check_some_url.py
that when run, will try to connect to the url provided. Save the file, update the MD5 sum and rebuild.