Add Flexible Replicate Software Type

Agenda

  • Prerequisite
  • Use request parameters to render a jinja template
  • Number of node depends on a parameter
  • Verification

In this tutorial we will introduce a new parameter replicate-quantity defining the quantity of nodes that we want for our HTML5AS CDN / replicate system.

This is made possible by making our replicate instance buildout file dynamic through Jinja2 templating. So far we only used the basic templating functionnalities, we will now automatically generate sections according to request parameters.

Here is the commit introducing this functionality to html5as Software release: commit diff.

Prerequisite

Use Request parameters to render replicate

Edit instance.cfg.in in order to the get the dict of parameters in the configuration key.

[slap-configuration]
...
key = ${slap-connection:key-file}
cert = ${slap-connection:cert-file}

Pass this configuration to the section [instance-replicate] rendering the replicate instance buildout.

[instance-replicate]
...
extensions = jinja2.ext.do
...
default-parameters =
  {
    "download_url": "",
    "replicate-quantity": 1
  }

Define replicate quantity from parameter

Edit instance_replicate.cfg.in to add this line at the beginning defining the replicate quantity from the replicate-quantity parameter. We set 1 as a default value for the parameter.

{%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
{%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}

Render Request sections dynamically

Edit instance_replicate.cfg.in to replace section [instance-1] and [instance-2] by a dynamic rendering.

...
# Set default title and port for each replicate based on requested quantity. Put this right after "{%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}".
{%- for i in range(1, replicate_quantity + 1) %}
{%- do parameter_dict.setdefault("title-%d" % i, "") %}
{%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
{%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
{%- endfor %}

It should look like this : 


{%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
{%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}
# Set default title and port for each replicate based on requested quantity
{%- for i in range(1, replicate_quantity + 1) %}
{%- do parameter_dict.setdefault("title-%d" % i, "") %}
{%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
{%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
{%- endfor %}

...

[instance-request-base]
...
# Remove : 
# config-title = {{ parameter_dict['title'] }} 
...
It should look like this : 


[instance-request-base]
<= slap-connection
recipe = slapos.cookbook:request
# It is the same software as the current one
software-url = ${slap-connection:software-release-url}
# We want the default behaviour
software-type = default
# What parameter are neede to be retrieved
return = server_url server-cdn-url monitor-setup-url
# Provided parameters
config-download_url = {{ parameter_dict['download_url'] }}

# Create request section in a loop.
{% for i in range(1, replicate_quantity + 1) %}
# Request a normal html5as instance
[instance-{{ i }}]
<= instance-request-base
# Name of the instance
name = instance-html5as-{{ i }}
config-port = {{ parameter_dict["port-%s" % i] }}
config-title = {{ parameter_dict["title-%s" % i] }}
config-monitor-httpd-port = {{ parameter_dict["monitor-httpd-port-%s" % i] }}
{% endfor %}

Don't forget to delete [instance-1] and [instance-2] (everything).

Have dynamic parameter and publishing

Edit instance_replicate.cfg.in to change existing [publish-connection-information] to define default parameters dynamically and publish node parameters dynamically.

# Publish information to connect to the two instances
[publish-connection-information]
recipe = slapos.cookbook:publish
{% for i in range(1, replicate_quantity + 1) %}
instance-{{ i }}-server_url = ${instance-{{ i }}:connection-server_url}
instance-{{ i }}-server-cdn-url = ${instance-{{ i }}:connection-server-cdn-url}
instance-{{ i }}-server-monitor-setup-url = ${instance-{{ i }}:connection-monitor-setup-url}
{% endfor %}

Check your changes

Check you changes in instance_replicate.cfg.in. The final file should be like this:

{%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
{%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}
# Set default title and port for each replicate based on requested quantity
{%- for i in range(1, replicate_quantity + 1) %}
{%- do parameter_dict.setdefault("title-%d" % i, "") %}
{%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
{%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
{%- endfor %}

 

# Standard buildout section
[buildout]
parts =
   publish-connection-information

 

eggs-directory = {{ buildout['eggs-directory'] }}
develop-eggs-directory = {{ buildout['develop-eggs-directory'] }}
offline = true

 

################################
# Sections to Request instances
################################

 

# Macro section sharing request parameters
[instance-request-base]
<= slap-connection
recipe = slapos.cookbook:request
# It is the same software as the current one
software-url = ${slap-connection:software-release-url}
# We want the default behaviour
software-type = default
# What parameter are neede to be retrieved
return = server_url server-cdn-url monitor-setup-url
# Provided parameters
config-download_url = {{ parameter_dict['download_url'] }}

 

# Create request section in a loop.
{% for i in range(1, replicate_quantity + 1) %}
# Request a normal html5as instance
[instance-{{ i }}]
<= instance-request-base
# Name of the instance
name = instance-html5as-{{ i }}
config-port = {{ parameter_dict["port-%s" % i] }}
config-title = {{ parameter_dict["title-%s" % i] }}
config-monitor-httpd-port = {{ parameter_dict["monitor-httpd-port-%s" % i] }}
{% endfor %}


 

# Publish information to connect to the two instances
[publish-connection-information]
recipe = slapos.cookbook:publish
{% for i in range(1, replicate_quantity + 1) %}
instance-{{ i }}-server_url = ${instance-{{ i }}:connection-server_url}
instance-{{ i }}-server-cdn-url = ${instance-{{ i }}:connection-server-cdn-url}
instance-{{ i }}-server-monitor-setup-url = ${instance-{{ i }}:connection-monitor-setup-url}
{% endfor %}

 

Update md5sums

Refer How To Move to md5sum automatic update to update md5sum:

$ cd ~/srv/project/slapos/software/html5as-base
$ ../../update-hash

Validation

  1. Request a replicate html5as with replicate-quantity=3
  2. Check connection parameter
 

Request a replicate html5as

To request a replicate type of sofware, you should define the type in the request script:

slapos request $software_name-1 $software_release_uri --type="replicate" --parameters replicate-quantity=3

Your request script should look like something like this  : 

#!/bin/sh
software_name=html5as-base
software_release_uri=~/srv/project/slapos/software/$software_name/software.cfg
slapos request $software_name'_1' $software_release_uri --type="replicate" --parameters replicate-quantity=3
title='John Doe' \
download_url='https://lab.nexedi.com/nexedi/converse.js/-/archive/nexedi-v4.2.0/converse.js-nexedi-v4.2.0.tar.gz' \
port=8086

Make sure to edit it in your request file instead of the example one. Parameters like replicate-quantity, title, etc. should place after --parameters directly as shown in the screenshot.

Run the request script by

$ cd ~/srv/project/slapos/software/html5as-base
$ bash ../../../request-html5as-base.sh 

Re-compile the software by

$ slapos node software --all

Once it completes, you may re-instantiate it by

$ slapos node instance --all

There is no errors expected. The compiling and instantiation should be completed.

Verify connection parameters

Make sure that the compiling and instantiation is completed. You can now inspect the connection parameters by re-running the request script:

$ cd ~/srv/project/slapos/software/html5as-base
$ bash ../../../request-html5as-base.sh 

As the replicate-quantity is 3, the published connection parameters will be 9 according to [publish-connection-information] defined in instance_replicate.cfg.in.

Thank You

  • Nexedi SA
  • 147 Rue du Ballon
  • 59110 La Madeleine
  • France
  • +33629024425

For more information, please contact Jean-Paul, CEO of Nexedi (+33 629 02 44 25).