in Programming, System Admin, Tips |

Glossary for SRE

This is a list of things for SRE, but mainly for myself. I easily forget terminologies and found myself look up terms even though I know what it is… (aging I guess?). So here’s the list of terms that I forget easily. (Not listing all the terms in SRE world since I know a lot of them already)

I don’t know why but I forget these so easily… >.<

  • Deployment Strategies
    • Canary Deployment
    • Rolling Deployment
    • Green Blue Deployment
  • MTTR – Mean Time To Repair
  • MTTF – Mean Time To Failure

Security alerts – github

First time I noticed Data Services on github repo and was so impressed with its Security alerts service. It’s free as of now, but I can kinda see where it may go… (cough cough blackduck… or a similar paid service)

Nonetheless, since I turned on Security alerts on some of my personal repos I have received so many emails warning about security vulnerabilities that exist in those repos. Also, there is another feature that github automatically submits a PR to fix the issue for you.

Code Hint annoyance and disable it

As a new programmer, I can see code hint being quite useful. However, as a seasoned programmer, those suggestions that Visual Studio Code pops up as I write code block the view of code that I am writing and I have to escape it so that I can see the rest of the code.

It is quite annoying and I’ve tried to find settings to disable it without googling and I lost. So I turned to google and found this stackoverflow thread.

Basically, need to add or update these in the user settings:

"editor.parameterHints.enabled": false,
"editor.suggest.snippetsPreventQuickSuggestions": false,
"html.suggest.html5": false,
"editor.snippetSuggestions": "none",

in Tips | 108 Words

Pact Broker setup with docker-compose

Almost all projects that I work with are set up through docker-compose by myself. This pact broker set up was no exception. There are so many advantages using docker-compose, specially on local environment.

Pact broker itself requires additional post, so I am not going to talk about it at all in this post…

These are the versions of docker and docker-compose respectively:

 macmini @ ~/projects/pact_broker * master
  [30] ? docker version
 Client:
  Version:           18.09.1
  API version:       1.39
  Go version:        go1.10.6
  Git commit:        4c52b90
  Built:             Wed Jan  9 19:35:23 2019
  OS/Arch:           linux/amd64
  Experimental:      false
 

 Server: Docker Engine - Community
  Engine:
   Version:          18.09.1
   API version:      1.39 (minimum version 1.12)
   Go version:       go1.10.6
   Git commit:       4c52b90
   Built:            Wed Jan  9 19:02:44 2019
   OS/Arch:          linux/amd64
   Experimental:     false
 

macmini @ ~/projects/pact_broker * master
  [31] ? docker-compose version
 docker-compose version 1.17.0, build ac53b73
 docker-py version: 2.5.1
 CPython version: 2.7.13
 OpenSSL version: OpenSSL 1.0.1t  3 May 2016

my pact broker project’s directory structure

docker-compose.yml content

version: '3.2'
services:
  postgresdb:
    build:
      context: ./db
      dockerfile: ./db/Dockerfile
    expose:
      - "5432"
    volumes:
      - ./db/data:/var/lib/postgresql/data
    environment:
      - PACTBROKER_USER_PASSWORD
      - POSTGRES_PASSWORD
      - POSTGRES_USER=admin
      - PGDATA=/var/lib/postgresql/data/pgdata
    restart: 'always'
  pactbroker:
    image: dius/pact-broker
    links:
      - "postgresdb:postgresdb"
    environment: 
       - "PACT_BROKER_DATABASE_PASSWORD=${PACTBROKER_USER_PASSWORD}"
       - "PACT_BROKER_DATABASE_USERNAME=pactbrokeruser"
       - "PACT_BROKER_DATABASE_HOST=postgresdb"
       - "PACT_BROKER_DATABASE_NAME=pactbroker" 
    ports:
      - "8081:80"
    restart: 'always'

One thing to note here is that I’ve hardcoded database name, database host, and pact broker’s database username since it’s local environment for development purpose. In production like environment, probably all of information can be pulled from environment rather than hardcoding them.

So on local environment, only information that you should set are PACTBROKER_USER_PASSWORD and POSTGRES_PASSWORD before running docker-compose up or docker-compose up -d.

Dockerfile file content

  [53] ? cat Dockerfile 
 ## Base image
 FROM postgres
 
 ## Of course, every Dockerfile requires a maintainer
 MAINTAINER me
 
 ## Add a special script to initialize db for pact broker
 ADD ./initial.sh /docker-entrypoint-initdb.d/

Dockerfile itself is pretty simple as shown above since only operation it has to do is to create a new database, new user, and grant the user full permission to the new database for pact broker and there is the entrypoint script that deals with that.

initial.sh file content

 #!/bin/bash
 set -e
 
 psql -v ON_ERROR_STOP=1 --username admin <<-EOSQL
   CREATE USER pactbrokeruser WITH PASSWORD '$PACTBROKER_USER_PASSWORD';
   CREATE DATABASE pactbroker WITH OWNER pactbrokeruser;
   GRANT ALL PRIVILEGES ON DATABASE pactbroker TO pactbrokeruser;
 EOSQL

That is it! The above script will be executed once when a new container is created via docker-compose and it will create a new user, a new database, and grant the user all privileges to the database.

Start Up!

Once those are in place, get back to the root of the project file and execute this:

docker-compose up

Only reason why I am not executing it in background mode is to see if there is any issue starting up the broker. Once confirming log stays calm, fire up your browser (chrome browser in my case), type 127.0.0.1:8081 in the browser, and witness the pact broker coming up on your browser.

The default port for the pact broker’s image is 80, but I had to change that to something else because I already had http server up and running at the port 80 and had to avoid the port conflict.

If you do not have anything up and running, you should change the port section in the docker-compose to “80:80” from “8081:80”

in Tips | 670 Words

Downgrade to openjdk 8 for Jenkins on Ubuntu 18 Bionic

On my Ubuntu 18 Bionic server, I installed Jenkins. Upon completion of the installation, there was error message that goes something like this:

Output from sudo journalctl -xe 

 Feb 19 14:10:17 macmini.shinstudio.lan jenkins[21270]: OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode)
 Feb 19 14:10:17 macmini.shinstudio.lan jenkins[21270]: Aborting
 Feb 19 14:10:17 macmini.shinstudio.lan systemd[1]: jenkins.service: Control process exited, code=exited status=1
 Feb 19 14:10:17 macmini.shinstudio.lan systemd[1]: jenkins.service: Failed with result 'exit-code'.
 Feb 19 14:10:17 macmini.shinstudio.lan systemd[1]: Failed to start LSB: Start Jenkins at boot time.
 -- Subject: Unit jenkins.service has failed
 -- Defined-By: systemd
 -- Support: http://www.ubuntu.com/support
 -- 
 -- Unit jenkins.service has failed.
 -- 
 -- The result is RESULT.
 Feb 19 14:10:18 macmini.shinstudio.lan systemd[1]: Reloading.
 Feb 19 14:10:18 macmini.shinstudio.lan systemd[1]: Starting resolvconf-pull-resolved.service...
 -- Subject: Unit resolvconf-pull-resolved.service has begun start-up
 -- Defined-By: systemd
 -- Support: http://www.ubuntu.com/support
 -- 
 -- Unit resolvconf-pull-resolved.service has begun starting up.
 Feb 19 14:10:18 macmini.shinstudio.lan systemd[1]: Started resolvconf-pull-resolved.service.
 -- Subject: Unit resolvconf-pull-resolved.service has finished start-up
 -- Defined-By: systemd
 -- Support: http://www.ubuntu.com/support
 -- 
 -- Unit resolvconf-pull-resolved.service has finished starting up.
 -- 
 -- The start-up result is RESULT.
 Feb 19 14:10:22 macmini.shinstudio.lan sudo[19303]: pam_unix(sudo:session): session closed for user root
 Feb 19 14:10:28 macmini.shinstudio.lan kernel: [UFW BLOCK] IN=enp0s10 OUT= MAC=01:00:5e:00:00:01:f0:9f:c2:c6:9e:f4:08:00 SRC=192.168.1.101 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 I
 Feb 19 14:11:19 macmini.shinstudio.lan sudo[21416]: shane : TTY=pts/0 ; PWD=/var/log/jenkins ; USER=root ; COMMAND=/bin/systemctl start jenkins
 Feb 19 14:11:19 macmini.shinstudio.lan sudo[21416]: pam_unix(sudo:session): session opened for user root by shane(uid=0)
 Feb 19 14:11:19 macmini.shinstudio.lan systemd[1]: Starting LSB: Start Jenkins at boot time...
 -- Subject: Unit jenkins.service has begun start-up
 -- Defined-By: systemd
 -- Support: http://www.ubuntu.com/support
 -- 
 -- Unit jenkins.service has begun starting up.
 Feb 19 14:11:19 macmini.shinstudio.lan jenkins[21419]: Found an incorrect Java version
 Feb 19 14:11:19 macmini.shinstudio.lan jenkins[21419]: Java version found:
 Feb 19 14:11:20 macmini.shinstudio.lan jenkins[21419]: openjdk version "10.0.2" 2018-07-17
 Feb 19 14:11:20 macmini.shinstudio.lan jenkins[21419]: OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)
 Feb 19 14:11:20 macmini.shinstudio.lan jenkins[21419]: OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode)
 Feb 19 14:11:20 macmini.shinstudio.lan jenkins[21419]: Aborting
 Feb 19 14:11:20 macmini.shinstudio.lan systemd[1]: jenkins.service: Control process exited, code=exited status=1
 Feb 19 14:11:20 macmini.shinstudio.lan sudo[21416]: pam_unix(sudo:session): session closed for user root
 Feb 19 14:11:20 macmini.shinstudio.lan systemd[1]: jenkins.service: Failed with result 'exit-code'.
 Feb 19 14:11:20 macmini.shinstudio.lan systemd[1]: Failed to start LSB: Start Jenkins at boot time.
 -- Subject: Unit jenkins.service has failed
 -- Defined-By: systemd
 -- Support: http://www.ubuntu.com/support
 -- 
 -- Unit jenkins.service has failed.
 -- 
 -- The result is RESULT.
 Feb 19 14:11:29 macmini.shinstudio.lan kernel: [UFW BLOCK] IN=enp0s10 OUT= MAC=01:00:5e:00:00:01:f0:9f:c2:c6:9e:f4:08:00 SRC=192.168.1.101 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 I
 Feb 19 14:11:33 macmini.shinstudio.lan sudo[21467]: shane : TTY=pts/0 ; PWD=/var/log/jenkins ; USER=root ; COMMAND=/bin/journalctl -xe
 Feb 19 14:11:33 macmini.shinstudio.lan sudo[21467]: pam_unix(sudo:session): session opened for user root by shane(uid=0)

The first thing I noticed was the Java version, which is 10.0.2+13-Ubuntu… In the past when I worked with Jenkins, JDK was mostly 8. So, I googled about JDK 10+ and Jenkins and found this link. I thought that the content was not convincing me much whether JDK 10+ would work well with Jenkins…

I decided to downgrade to JDK 8 and here’s what I did (did not actual uninstall JDK 10 and reinstall JDK 8)

sudo update-alternatives –config java
There are 2 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
————————————————————
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode

Press to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in manual mode

[20] ? java -version
openjdk version “1.8.0_191”
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

After confirming java executable points to 1.8 instead of 10/11, I started Jenkins successfully.

[21] ? sudo systemctl start jenkins 
[22] ? ps aux | grep jenkins
 jenkins  21545  0.2  0.2  76632  7640 ?        Ss   14:14   0:00 /lib/systemd/systemd --user
 jenkins  21546  0.0  0.0 257560  2936 ?        S    14:14   0:00 (sd-pam)
 jenkins  21561  0.0  0.0  19096   188 ?        S    14:14   0:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
 jenkins  21562  100  3.0 2953332 114708 ?      Sl   14:14   0:13 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
 shane    21617  0.0  0.0  13136  1088 pts/0    S+   14:14   0:00 grep --color=auto jenkins

No error and Jenkins is up and running!

Writing Testable Code in Go

I’ve had a hard time to wrap around the stub, mock in golang, but this blog post helped me immensely.

Link here to view the post

After reading it, my biggest issue was lack of my own interfaces in my program. Without my own interface, it was a big challenge to mock library defined interfaces; thus, making it quite hard to come up with tests.

After writing test, execute it like this:

 

RancherOS

Recently I got to know about RancherOS, which is an OS that provides pretty much only Docker host environment so that you can launch Docker containers.

My test environment is Mac OS Sierra with VMWare Fusion 8.5. What I did was as follows:

  1. Download ISO file for RancherOS
  2. Launch ISO via VMWare Fusion with 1GB ram and 10GB disk space (That’s normally what I use for linux distribution guest environment)
  3. Once I was on RancherOS terminal, I created cloud-config.yml file with ssh publick key in it. (Refer to the instruction)
  4. I made a note of IP address for the step #5
  5. After installing to the disk, I chose to reboot. From Mac terminal, I ssh’ed to the IP address of the RancherOS guest after RancherOS guest became available.

Pretty straightforward to install RancherOS on my mac.

RancherOS provides its own command line tool called ‘ros’ and this is the tool to configure the system, docker, etc.

Sync up Docker container time with Host’s

I’ve had some issues with my application’s user session being expired as soon as it got created. After further investigation, it was due to the fact that docker container’s time was far behind the current PDT(which is where I am and a default timezone for all my servers).

Easiest solution was to mount these two files from host to any containers:

After the two lines of docker volume, viola. Time synced between host and container.

Bye bye data-only container! Hello named volume!

Persistent data practice that I’ve been using with Docker has been the host volume although I knew that data-only container pattern existed.

So this time around, I was going to start utilizing data-only container pattern for some of db containers. However, I got to know that after docker version 1.9.0.

For example with data-only container pattern, we would have a docker-compose file like this:

However, a better approach would be: