systemd and docker-compose

Overview::

docker compose

A short how to on integrating docker-compose with systemd. Advantage of this is ,you can have your containers setup with docker-compose to start up on boot or get managed with systemd itself.

Howto::

Create a new file docker-compose.service at /usr/lib/systemd/system/

Contents of the file should be as below.

[Unit]
Description=docker-compose
Requires=docker.service
After=docker.service

[Service]
Restart=always

# Remove old containers, images and volumes
ExecStartPre=/usr/bin/docker-compose -f your_docker_compose_file.yml down -v
ExecStartPre=/usr/bin/docker-compose -f your_docker_compose_file.yml rm -v
ExecStartPre=-/bin/bash -c 'docker volume rm $(docker volume ls -q)'
ExecStartPre=-/bin/bash -c 'docker rmi $(docker images | grep "<none>" | awk \'{print $3}\')'
ExecStartPre=-/bin/bash -c 'docker rm -v $(docker ps -aq)'

# Compose up
ExecStart=/usr/bin/docker-compose -f your_docker_compose_file.yml up

# Compose down, remove containers and volumes
ExecStop=/usr/bin/docker-compose -f your_docker_compose_file.yml down -v

[Install]
WantedBy=multi-user.target

You can now manage your containers with the below commands

systemctl start docker-compose
systemctl stop docker-compose

Make sure the docker containers are up and available with

docker-compose ps

To start your containers setup with docker compose to start on system boot, enable it on systemd

systemctl enable docker-compose

Reference  ::
https://github.com/docker/compose/issues/4266

LEMP : Nginx php-fpm mariadb with docker compose

Overview::

docker compose

Dockerising, a LEMP stack ( Nginx Php-fpm and Mariadb ) with docker-compose on Linux . Please check out this post to see how the same can be done with docker command line .

Installation::

Installation of docker-compose is covered in detail on official site,  do read through it . Pasting the quick howto  for debian here.

sudo apt-get install python-pip
sudo pip install --upgrade pip
sudo pip install docker-compose

Howto ::

  1. If you are following the last post you should now have the directory structure and conf files as below
    |-- mariadb
    | |-- data
    | | |-- ...
    | | `-- ...
    |-- nginx
    | |-- certs
    | |-- conf.d
    | | `-- techblog.com
    | |-- files
    | | |-- ...
    | | `-- ...
    | |-- log
    | | |-- ...
    | | `-- ...
    `-- phpfpm 
     |-- Dockerfile 
    
    
  2. Create  docker-compose.yml
    Create the docker-compose.yml file as below

    version: '3'
    services:
     mariadb:
     image: mariadb:latest
     volumes:
     - ./mariadb/data:/var/lib/mysql
     env_file:
     - .env
     phpfpm:
     build: ./phpfpm/
     volumes:
     - ./nginx/files:/home/tech
     nginx:
     image: nginx:alpine
     volumes:
     - ./nginx/conf.d:/etc/nginx/conf.d 
     - ./nginx/certs:/etc/nginx/certs 
     - ./nginx/log:/var/log/nginx
     - ./nginx/files:/home/tech
     ports:
     - 80:80
     - 443:443

    First line mentions the version of docker-compose syntax we are using. The later lines mentions the services and options they use, you can cross-reference them with the  docker commands used in the first post

  3. Make the LEMP stack online.
    docker-compose up -d 

    Above command will bring the 3 services mariadb, phpfpm and nginx online by deploying the respective  containers . Phpfpm container will be deployed after building the image with Dockerfile, rest of the services will be containerised from the base images. A bridge network will be auto created by the docker-compose and all the services will be added to it so that they can talk to each other.

  4. Monitoring/ Troubleshooting
    docker-compose ps 
    docker-compose top
    docker-compose logs
  5. Making the stack offline
    docker-compose down
    

LEMP : Nginx php-fpm mariadb with docker

docker commandlineOverview::

Dockerising, a LEMP stack ( Nginx Php-fpm and Mariadb ) with command line on Linux . As a real world example we’ll be restoring a wordpress site to this stack .

Installation::

Installation of docker is covered in detail on official site, please do read through it . Pasting the quick howto  for debian here.

sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"

sudo apt-get install docker-ce

Howto ::

  1. Create the directory structure as below  /root/lemp/
    |-- mariadb
    | |-- data
    |-- nginx
    | |-- certs
    | |-- conf.d
    | |-- files
    | `-- log
    `-- phpfpm
    
  2. Create a network
    Below command creates a docker bridge network named internal

    docker create network internal
  3. Deploy Mariadb container .
    docker run --name mariadb --network=internal -v /root/lemp/mariadb/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=your_secret_pass  -d mariadb:latest

    Above command will, download the latest mariadb image.
    run : Run it with the name mariadb.
    -v : Create a volume mount for /var/lib/mysql to host dir /root/lemp/mariadb/data.
    -e :Pass the mysql root password as environment variable.
    —network : Attach the container to bridge network internal.
    -d :Detach  and run the container in background.

  4. Create DB/grant privileges/restore database
    docker exec -i mariadb mysql -uroot -pyour_secret_pass -e "create database tech_blog"
    docker exec -i mariadb mysql -uroot -pyour_secret_pass -e "GRANT ALL PRIVILEGES ON techie_blog.* TO 'techie_user'@'%' identified by 'your_db_pass'"
    docker exec -i mariadb mysql -uroot -pyour_secret_pass --force tech_blog < mariadb/tech_blog.sql

    exec -i : executes commands interactively in a running container

  5. Enable mysqli support on php-fpm.
    As we are hosting a WP site on this stack we need to have mysqli support enabled for php-fpm, this can be achieved by creating a new image with mysqli support

    cd /root/phpfpm
    vi  Dockerfile

    FROM php:fpm-alpine
    RUN docker-php-ext-install mysqli

    
    docker build -t php_fpm .
    

    build -t : builds a new image and tags it as php_fpm:latest

  6. Deploy Php-fpm
    docker run -d --name phpfpm --network=internal -v /root/lemp/nginx/files:/home/tech  -v /root/lemp/phpfpm/php-fpm.d:/etc/php-fpm.d/  php_fpm:latest 
  7. Configure nginx
    Copy the WP site nginx conf file to /root/lemp/nginx/conf.d/techblog.com

    cat /root/lemp/nginx/conf.d/techblog.com

    server {
    listen 80;
    server_name techblog.com;
    root /home/tech ;
    access_log /var/log/nginx/techblog.com;
    index index.php index.html index.htm;
    location / {
    try_files $uri $uri/ /index.php$args;
    }error_page 404 /404.html;error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    root /home/tech;
    }

    # pass the PHP scripts to FastCGI server listening on the php-fpm socket
    location ~ \.php$ {
    try_files $uri =404;
    fastcgi_pass phpfpm:9000;
    fastcgi_index index.php;
    fastcgi_split_path_info ^(/)(/.*)$;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;

    }

  8. WP file copy/configure
    Copy the WP files to /root/lemp/nginx/files
    Change database host configuration in the wp-config.php as below to point it to the Mariadb container

    define(‘DB_HOST’, ‘mariadb’);

  9. Deploy Nginx container
    docker run -d -p 80:80 --name nginx --network=internal -v /root/lamp/nginx/conf.d:/etc/nginx/conf.d -v /root/lamp/nginx/certs:/etc/nginx/certs -v /root/lamp/nginx/log:/var/log/nginx -v /home/tech:/home/tech nginx:alpine

    -p : exposes the nginx port 80 of container and binds it to port 80 on the host.

Troubleshooting::

docker ps  # Shows the running containers
docker logs <container_id> # shows the container logs
docker inspect <container_id>

Agri IoT :: Prototype 3 : Overview & Actuator

Overview::

With prototype 3 of Agri IoT, attempt is to proceed further by creating a actuator node that can be controlled from Openhab. Pro mini 3,3v 8Mhz is selected because of its small form, low power consumption and NRF24L01+ will work with it as a radio. A 4 CH 5V relay module is used to switch on/off devices and to power the whole setup a breadboard 3.3v/5v power supply is used Mysensors v2.0 API is used to build and connect the blocks.  Prototype 3 actuator node setup is covered in this article, rest is in Agri IoT :: Prototype 2 : Gateway & Controller 

Prototype 1&2 is covered in below articles,

  1. Agri IoT :: Prototype 1 : Overview
  2. Agri IoT :: Prototype 2 : Overview

Objective ::

Actuator node should,

  • Be able to switch on/off the supply to devices according to input from Openhab2 controller

Components ::

  • Arduino pro mini 3.3v 8Mhz
  • NRF24L01+
  • CP2102 USB to TTL Serial
  • Relay module 5v
  • Breadboard power supply 3.3v/5v

How to ::

Idea is to power on actuator in the node with one of the Arduino digital out,

Hardware:

Relay

  • 1st relay input should be connected to the digital pin 3 of Arduino pro mini.
  • Relay power should be provided by 5v ourput of breadboard power supply.

Radio:

  • Connection should be made as described here
ARDUINO NRF24L01+
GND GND
5VReg
3.3V
VCC
9 CE
10 CSN/CS
13 SCK
11 MOSI
12 MISO
2 IRQ
  • Power to arduino and radio should be provided by 3.3v output of breadboard powersupply

Software:

Connect the arduino via Mini usb cable. Make sure you select the right options as mentioned here

Upload the sketch to actuator node. If you want to manually set the node ID specific the below line above #include <Mysensor.h>

#define MY_NODE_ID 2 // node id 
#include <MySensors.h>

Testing & Troubleshooting::

Use Arduino IDE serial monitor and set the baud rate to 115200.  You can find sensor node related debug output there.

Reference :: 
https://www.mysensors.org/build/connect_radio
https://www.mysensors.org/build/relay
https://www.arduino.cc/en/Main/ArduinoBoardProMini
http://www.instructables.com/id/NRF24L01-Fixing-Connection-Issues/
https://forum.mysensors.org/topic/5174/solved-non-healing-tsm-msg-len-error-hours-into-run/8

Update : 21/05/2017

Lessons learned the hardway : If your actuator node is disconnecting and staying of the grid it can be because of the noise from the relay module. The message similar to below in your node serial output denotes this.

TSF:MSG:READ,76-76-76,s=76,c=4,t=76,pt=2,l=9,sg=1:19532
TSF:MSG:LEN,20!=32

Things that helped me to fix this

  • Check the earth line of your electrical connections with a tester and make sure there is no current leakage and its well grounded
  • Connect the relay module to a separate 5v DC power supply, make sure you have the ground lines connected across
  • Move the relay board away from the arduino/radio
  • Add a decoupling capacitor to the radio as described here . This helps to smooth en the spikes if any
  • Add the below line to your arduino code before #include <MySensors.h> . This will help in resetting the radio in case of connection issues.
#define MY_TRANSPORT_SANITY_CHECK

Agri IoT :: Prototype 2 : Gateway & Controller

Overview::

With prototype 2 of Agri IoT, attempt is to proceed further by adding persistence to the OpenHab2 controller, charting of sensor data is also done. Mysensors v2.0 API is used to build and connect the blocks.  Prototype 2 gateway and controller is covered in this article, check out Agri IoT :: Prototype 2 : Overview & Sensor 

Prototype 1 is covered in below articles,

  1. Agri IoT :: Prototype 1 : Overview
  2. Agri IoT :: Prototype 1 : Sensor
  3. Agri IoT :: Prototype 1 : Gateway
  4. Agri IoT :: Prototype 1 : Controller

Objective ::

  • No changes in gateway same as on prototype 1
  • Controller should be able to store the sensor data
  • Controller will be plotting the data in charts.

Components::

  • PC running Debian Jessie
  • NodeMUC
  • NRF240L+

Howto::

Openhab2 installation  will be configured to have a persistent storage and sitemap/items configuration will be modified to show charts.

  • Install influxdb persistance

Access http://<controller_ip>:8080/paperui/index.html#/extensions under Persistence  >> InfluxDB (v 1.0) Persistence >> Install

Make sure the installation is properly done with

# ssh openhab@localhost -p 8101
Default pass : habopen

openhab> log:tail

  • install influxdb and setup a database


# curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
# source /etc/os-release
# test $VERSION_ID = "8" && echo "deb https://repos.influxdata.com/debian jessie stable" | tee /etc/apt/sources.list.d/influxdb.list
# apt-get update && sudo apt-get install influxdb
# systemctl start influxdb
# sudo systemctl enable influxdb

$ influx
Connected to http://localhost:8086 version 0.13
InfluxDB shell version: 0.13
> CREATE DATABASE openhab_db
> CREATE USER admin WITH PASSWORD 'new_admin_pass' WITH ALL PRIVILEGES
> CREATE USER openhab WITH PASSWORD 'new_hab_pass'
> GRANT ALL ON openhab_db TO openhab
> exit
  • Configure influxdb

Influxdb should be configured as below. Below configuration logs all sensor data into a database called openhab_db that was created on previous step

Enable http auth

# vi /etc/influxdb/influxdb.conf

[http] enabled = true
bind-address = “:8086” # change to a specific interface if needed
auth-enabled = true # will enforce authentication

# cat /etc/openhab2/persistence/influxdb.persist

Strategies {
   everyHour : "0 0 * * * ?"
}
Items {
 * : strategy =  everyChange, everyHour
}

# cat /etc/openhab2/services/influxdb.cfg

# The database URL, e.g. http://127.0.0.1:8086 or https://127.0.0.1:8084 .
# Defaults to: http://127.0.0.1:8086
url=http://<controller_ip>:8086

# The name of the database user, e.g. openhab.
# Defaults to: openhab
user=openhab

# The password of the database user.
password=new_hab_pass

# The name of the database, e.g. openhab.
# Defaults to: openhab
db=openhab_db

# The retention policy to be used, needs to configured in InfluxDB
# Till v0.13: 'default', since v1.0: 'autogen'
retentionPolicy=autogen
  • Configure multiple sensors and charts

Below item and sitemap configuration is done to show data collected from multiple sensors on the sensor node. 5 charts are also configured.  First chart showing the grouped data with ability to dynamically switch the time period of plotting and rest with hourly data plotted for each sensors.

# cat /etc/openhab2/items/agri.items

Group SoilSensors

Number SoilSensors_Period "Viewing interval" <chart>
Number Agri_moist "Moisture [%.1f ]" <grass> (SoilSensors) {mqtt="<[broker:mygateway1-out/1/1/1/0/37:state:default]"}
Number Agri_temp "Temperature [%.1f °C]" <temperature> (SoilSensors) {mqtt="<[broker:mygateway1-out/1/3/1/0/0:state:default]"}
Number Agri_hum "Humidity [%.1f ]" <humidity> (SoilSensors) {mqtt="<[broker:mygateway1-out/1/4/1/0/1:state:default]"}
Number Agri_lux "Illuminance [%.1f lux]" <light> (SoilSensors) {mqtt="<[broker:mygateway1-out/1/2/1/0/23:state:default]"}

# cat /etc/openhab2/sitemaps/agri.sitemap

sitemap demo label="Main Menu"
{
Frame label="Agri Sensor" {
Text item=Agri_moist
Text item=Agri_temp
Text item=Agri_hum
Text item=Agri_lux
Switch item=SoilSensors_Period label="Chart Period" mappings=[0="Hour", 1="Day", 2="Week"]
Chart item=SoilSensors period=h refresh=300 visibility=[SoilSensors_Period==0, SoilSensors_Period=="Uninitialized"]
Chart item=SoilSensors period=D refresh=1800 visibility=[SoilSensors_Period==1]
Chart item=SoilSensors period=W refresh=3600 visibility=[SoilSensors_Period==2]
}
}

You would be able to see the sensor output and plotted graph here

http://:8080/basicui/app?sitemap=agri

Testing & Troubleshooting::

Logs related to openhab2 can be viewed in Karaf console.


# ssh openhab@localhost -p 8101
Default pass : habopen

openhab> logs:tail

References :: 
http://www.homeautomationforgeeks.com/openhab_persistence.shtml
https://community.openhab.org/t/charting-in-oh2/10410
https://community.openhab.org/t/influxdb-grafana-persistence-and-graphing/13761