SlapOS Home SlapOS

    How To Add Flexible Replicate Software Type

    How to add flexible replicate software type
    • Last Update:2023-08-18
    • Version:004
    • Language:en

    Agenda

    • Prerequisite
    • Use request parameters to render a jinja template
    • Number of node depends on a parameter
    • Verification

    In this tutorial we will introduce a new parameter replicate-quantity defining the quantity of nodes that we want for our HTML5AS CDN / replicate system.

    This is made possible by making our replicate instance buildout file dynamic through Jinja2 templating. So far we only used the basic templating functionnalities, we will now automatically generate sections according to request parameters.

    Here is the commit introducing this functionality to html5as Software release: commit diff.

    Prerequisite

    Use Request parameters to render replicate

    Edit instance.cfg.in in order to the get the dict of parameters in the configuration key.

    [slap-configuration]
    ...
    key = ${slap-connection:key-file}
    cert = ${slap-connection:cert-file}
    

    Pass this configuration to the section [instance-replicate] rendering the replicate instance buildout.

    [instance-replicate]
    ...
    extensions = jinja2.ext.do
    ...
    default-parameters =
      {
        "download_url": "",
        "replicate-quantity": 1
      }

    Define replicate quantity from parameter

    Edit instance_replicate.cfg.in to add this line at the beginning defining the replicate quantity from the replicate-quantity parameter. We set 1 as a default value for the parameter.

    {%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
    {%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}
    

    Render Request sections dynamically

    Edit instance_replicate.cfg.in to replace section [instance-1] and [instance-2] by a dynamic rendering.

    ...
    # Set default title and port for each replicate based on requested quantity. Put this right after "{%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}".
    {%- for i in range(1, replicate_quantity + 1) %}
    {%- do parameter_dict.setdefault("title-%d" % i, "") %}
    {%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
    {%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
    {%- endfor %}
    
    It should look like this : 
    
    
    {%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
    {%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}
    # Set default title and port for each replicate based on requested quantity
    {%- for i in range(1, replicate_quantity + 1) %}
    {%- do parameter_dict.setdefault("title-%d" % i, "") %}
    {%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
    {%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
    {%- endfor %}
    
    ...
    
    [instance-request-base]
    ...
    # Remove : 
    # config-title = {{ parameter_dict['title'] }} 
    ...
    It should look like this : 
    
    
    [instance-request-base]
    <= slap-connection
    recipe = slapos.cookbook:request
    # It is the same software as the current one
    software-url = ${slap-connection:software-release-url}
    # We want the default behaviour
    software-type = default
    # What parameter are neede to be retrieved
    return = server_url server-cdn-url monitor-setup-url
    # Provided parameters
    config-download_url = {{ parameter_dict['download_url'] }}
    
    # Create request section in a loop.
    {% for i in range(1, replicate_quantity + 1) %}
    # Request a normal html5as instance
    [instance-{{ i }}]
    <= instance-request-base
    # Name of the instance
    name = instance-html5as-{{ i }}
    config-port = {{ parameter_dict["port-%s" % i] }}
    config-title = {{ parameter_dict["title-%s" % i] }}
    config-monitor-httpd-port = {{ parameter_dict["monitor-httpd-port-%s" % i] }}
    {% endfor %}

    Don't forget to delete [instance-1] and [instance-2] (everything).

    Have dynamic parameter and publishing

    Edit instance_replicate.cfg.in to change existing [publish-connection-information] to define default parameters dynamically and publish node parameters dynamically.

    # Publish information to connect to the two instances
    [publish-connection-information]
    recipe = slapos.cookbook:publish
    {% for i in range(1, replicate_quantity + 1) %}
    instance-{{ i }}-server_url = ${instance-{{ i }}:connection-server_url}
    instance-{{ i }}-server-cdn-url = ${instance-{{ i }}:connection-server-cdn-url}
    instance-{{ i }}-server-monitor-setup-url = ${instance-{{ i }}:connection-monitor-setup-url}
    {% endfor %}
    

    Check your changes

    Check you changes in instance_replicate.cfg.in. The final file should be like this:

    {%- set parameter_dict = dict(default_parameter_dict, **slapparameter_dict) %}
    {%- set replicate_quantity = int(parameter_dict['replicate-quantity']) %}
    # Set default title and port for each replicate based on requested quantity
    {%- for i in range(1, replicate_quantity + 1) %}
    {%- do parameter_dict.setdefault("title-%d" % i, "") %}
    {%- do parameter_dict.setdefault("port-%d" % i, 8081 + i) %}
    {%- do parameter_dict.setdefault("monitor-httpd-port-%d" % i, 8197 + i) %}
    {%- endfor %}

     

    # Standard buildout section
    [buildout]
    parts =
       publish-connection-information
    

     

    eggs-directory = {{ buildout['eggs-directory'] }}
    develop-eggs-directory = {{ buildout['develop-eggs-directory'] }}
    offline = true
    

     

    ################################
    # Sections to Request instances
    ################################

     

    # Macro section sharing request parameters
    [instance-request-base]
    <= slap-connection
    recipe = slapos.cookbook:request
    # It is the same software as the current one
    software-url = ${slap-connection:software-release-url}
    # We want the default behaviour
    software-type = default
    # What parameter are neede to be retrieved
    return = server_url server-cdn-url monitor-setup-url
    # Provided parameters
    config-download_url = {{ parameter_dict['download_url'] }}

     

    # Create request section in a loop.
    {% for i in range(1, replicate_quantity + 1) %}
    # Request a normal html5as instance
    [instance-{{ i }}]
    <= instance-request-base
    # Name of the instance
    name = instance-html5as-{{ i }}
    config-port = {{ parameter_dict["port-%s" % i] }}
    config-title = {{ parameter_dict["title-%s" % i] }}
    config-monitor-httpd-port = {{ parameter_dict["monitor-httpd-port-%s" % i] }}
    {% endfor %}


     

    # Publish information to connect to the two instances
    [publish-connection-information]
    recipe = slapos.cookbook:publish
    {% for i in range(1, replicate_quantity + 1) %}
    instance-{{ i }}-server_url = ${instance-{{ i }}:connection-server_url}
    instance-{{ i }}-server-cdn-url = ${instance-{{ i }}:connection-server-cdn-url}
    instance-{{ i }}-server-monitor-setup-url = ${instance-{{ i }}:connection-monitor-setup-url}
    {% endfor %}

     

    Update md5sums

    Refer How To Move to md5sum automatic update to update md5sum:

    $ cd ~/srv/project/slapos/software/html5as-base
    $ ../../update-hash

    Validation

    1. Request a replicate html5as with replicate-quantity=3
    2. Check connection parameter
     

    Request a replicate html5as

    To request a replicate type of sofware, you should define the type in the request script:

    slapos request $software_name-1 $software_release_uri --type="replicate" --parameters replicate-quantity=3

    Your request script should look like something like this  : 

    #!/bin/sh
    software_name=html5as-base
    software_release_uri=~/srv/project/slapos/software/$software_name/software.cfg
    slapos request $software_name'_1' $software_release_uri --type="replicate" --parameters replicate-quantity=3
    title='John Doe' \
    download_url='https://lab.nexedi.com/nexedi/converse.js/-/archive/nexedi-v4.2.0/converse.js-nexedi-v4.2.0.tar.gz' \
    port=8086

    Make sure to edit it in your request file instead of the example one. Parameters like replicate-quantity, title, etc. should place after --parameters directly as shown in the screenshot.

    Run the request script by

    $ cd ~/srv/project/slapos/software/html5as-base
    $ bash ../../../request-html5as-base.sh 

    Re-compile the software by

    $ slapos node software --all

    Once it completes, you may re-instantiate it by

    $ slapos node instance --all

    There is no errors expected. The compiling and instantiation should be completed.

    Verify connection parameters

    Make sure that the compiling and instantiation is completed. You can now inspect the connection parameters by re-running the request script:

    $ cd ~/srv/project/slapos/software/html5as-base
    $ bash ../../../request-html5as-base.sh 
    

    As the replicate-quantity is 3, the published connection parameters will be 9 according to [publish-connection-information] defined in instance_replicate.cfg.in.