Tiny struggles - engineering blog

Notifications on the channels in Golang

Sun, 09 Jul 2017 00:00:00 +0100

Notification on a channel is a common concurrency pattern in Golang. In this post I'm describing how it works and a common trap that I found with it.

Introduction to the pattern

This pattern can be used in a situation when there is some piece of work that updates a variable that then can be consumed by multiple other threads interested in this value. Imagine a family: mum is cooking a dinner, she finishes, announces to the children that the dinner is ready and the children after hearing it come and eat the dinner.

How could such a thing be implemented in Golang?

I'm assuming you are familiar with the golang channels and the closing of channels.

To implement such notification system, start with a struct:

with a variable that will be read,
with a channel for the notification that the value is ready.

When the value is updated, the producer thread closes the associated variableChannel. When the channel gets closed all the reading operations from it finish, so effectively the consumer threads will be notified that the thing that they were waiting for finished.

As an illustration, take a look at the Foo struct that comes up with a number that can later be read by 10 different goroutines when it's ready (full example also at https://play.golang.org/p/tolS8Qgeja).

type Foo struct {
    number     int
    numberChan chan struct{}
}

func (f *Foo) ComeUpWithANumber() {
    time.Sleep(1 * time.Second)
    f.number = 42
    close(f.numberChan)
}

func (f *Foo) Number() (int, error) {
    <-f.numberChan
    return f.number, nil
}

func main() {
    f := &Foo{numberChan: make(chan struct{})}
    go f.ComeUpWithANumber()
    for i := 0; i < 10; i++ {
        go func() {
            n, err := f.Number()
            if err != nil {
                fmt.Println("Error:", err)
            } else {
                fmt.Println("Number was", n)
            }
        }()
    }
    time.Sleep(2 * time.Second)
}

When you run it the result is as expected:

Number was 42
Number was 42
Number was 42
Number was 42
Number was 42
Number was 42
Number was 42
Number was 42
Number was 42
Number was 42

Ordering trap

Cool! The code works as expected. However, imagine that after some time users of your software start complaining that sometimes Foo spends too much time generating the number. They request a cancellation and timeout feature from you.

Sure no problem, you add the feature. Accessing the number becomes a bit more complex (full example at: https://play.golang.org/p/9JR-OONNsH).

func (f *Foo) Number() (int, error) {
    select {
    case <-f.doneChan:
        return 0, errors.New("canceled")
    case <-f.numberChan:
        return f.number, nil
    }
}

You test your code and you get the unexpected result. There is a bug there!

Error: canceled
Number was 42
Error: canceled
Error: canceled
Number was 42
Number was 42
Number was 42
Error: canceled
Error: canceled
Error: canceled

This result is a surprise for you because you know that there was no timeout here and that you close the doneChan after you've updated the number in your testing scenario. Just like below:

func (f *Foo) ComeUpWithANumber() {
    time.Sleep(1 * time.Second)
    f.number = 42
    close(f.numberChan)
    close(f.doneChan)
}

What happened? The answer is that the select clause will choose any of the "ready" channels, not necessarily the one that got ready first. There is no assured order.

So, even if the number is ready, before doneChan was closed, you might end up on the doneChan part of select.

To fix this issue, don't assume the order of accessing the closed channels. Be ready for any possible ordering. The improved version of the Number function looks as follows:

func (f *Foo) Number() (int, error) {
    select {
    case <-f.doneChan:
        select {
        case <-f.numberChan:
            return f.number, nil
        default:
            return 0, errors.New("canceled")
        }
    case <-f.numberChan:
        return f.number, nil
    }
}

Real life example

This all might seem a bit far fetched to you. But sadly it isn't. Here is an example from the golang grpc implementation.

The original code in the implementatin of the Stream struct was using select on multiple channels:

select {
case <-s.ctx.Done():
    return nil, ContextErr(s.ctx.Err())
case <-s.goAway:
    return nil, ErrStreamDrain
case <-s.headerChan:
    return s.header.Copy(), nil
}

And sometimes you would receive the header value and sometimes you would not...

Golang Concurrency Exercises

Wed, 21 Oct 2015 00:00:00 +0100

First, little disclaimer: concurrency is hard. Even though, someone can tell you that with a great support for concurrency primitives in Go concurrency will become easy. Trivial things will be easy, but you can still end up having problems.

If you are still feeling adventurous, I have some resources for you!

Some time ago, I organized Concurrent programming in Go workshop in Warsaw hackerspace. Slides.

People taking part in the workshop were Go beginners and most of them didn't have much experience in writing concurrent programs. Workshops were interactive, so I have some exercises for you. Exercises start with number 3, because I've omitted the trivial ones (you can still read about them in the slides). They would require you to use your knowledge about concurrency and software design, expect them to be open ended.

Exercise 3 - vending machine

Design a vending machine that will take coins as input and chocolate as output.

Let's imagine that Gophers have coins of value 1, 2 and 5. Gopher can choose the item and then has to insert money and wait for the vending machine to dispense the item.

Exercise 4 - design simple work pipeline

Hackers started working on the new IRC bot. They decided that the bot should support command: ~shoutitinreverse

Your task is to design a following string processing pipeline: put on queue -> strip whitespace -> change to UPPER -> reverse -> print

What additional features could it support?

Exercise 5 - control access to 3D printers

During a busy hackaton 3D printers are heavily used. Write a simulation of hackers trying to access 3D printers.

In the hackerspace, there are 3 3D printers. There are 7 hackers that are interested in using the printers.

If the hacker can't access the printer for more than 5 seconds, he gets annoyed and quits the hackaton. Hackers use printers for random interval from 1 to 10 seconds and usually they need to use the printer at least twice, because nothing is perfect for the first time.

Exercise 6 - hackerspace membership fees

Every month, hackerspace has to pay its bills and every hacker has to pay his or her membership fee.

Requirements: - Hackers in general don't pay their fees the same day every month, it's much more random. - Bills however have to be paid on time. - The board wants monthly reports of the funds.

Use share memory by communication approach (avoid mutexes).

Summary

I hope that it would be useful. If you finish those exercises I would love to see your solutions. Exercises were inspired by 3 TRIVIAL CONCURRENCY EXERCISES FOR THE CONFUSED NEWBIE GOPHER.

Have fun learning Go!

Golang Code Reuse - Generalization tricks

Sat, 29 Aug 2015 00:00:00 +0100

In this article I want to share with you few tricks that I learned in go that would allow you to write reusable code. I will start with code that is written for only one use case and refactor it to be reusable.

You can see all code from this blogpost in github repo.

How to reuse code in Go?

If you want something similar that you already have, what should you do? How do people avoid duplication without inheritance and generics?

TL;DR - Using golang interfaces and composition.

Instead of inheritance, you should use composition and accept interfaces instead of concrete types. Instead of generics, use narrow interfaces describing exactly what you need or if there is no other way and you want to be very generic - use empty interfaces interface{} with unboxing.

If you feel little uneasy about golang interfaces, I recommend you to read some tutorials first:

Algorithmic example

I was recently reading section about backtracking in The Algorithm Design Manual by Skiena. Backtracking is a very useful algorithm that can be used to solve puzzles, generate all possible permutations of given set or all it subsets.

Sudoku puzzle being solved by backtracking

In the book author describes there universal backtracking algorithm. Examples in the book are written in C and I decided that I would rewrite them to Go.

Core of alorithm is a Backtrack function, it uses problem specific functions:

IsSolution(a []elements, k int, n int)
ConstructCandidates(a []elements, k int, n int) []elements
ProcessSolution(a []elements, k int, n int)

To solve your problem, you just need to substitute those functions.

I succeeded in translating the algorithm to Go for the subset generation problem, you can see the code below.

package main

import (
    "fmt"
)

var total int

func Backtrack(a []bool, k int, input int) {
    if IsSolution(a, k, input) {
        ProcessSolution(a, k, input)
        return
    }
    k++

    candidates := ConstructCandidates(a, k, input)
    for _, c := range candidates {
        a[k-1] = c
        Backtrack(a, k, input)
    }
}

func IsSolution(a []bool, k int, n int) bool {
    return k == n
}

func ConstructCandidates(a []bool, k int, n int) []bool {
    return []bool{true, false}
}

func ProcessSolution(a []bool, k int, n int) {
    fmt.Printf("{")
    for i, v := range a {
        if v {
            fmt.Printf("%d", i)
        }
    }
    fmt.Printf("}\n")
    total++
}

func main() {
    a := make([]bool, 10)
    Backtrack(a, 0, 10)
    fmt.Println("total", total)
}

If you run the script it will print all possible subsets of range from 0 to 9 inclusive.

$  go run simplebacktrack.go 

{0123456789}
{012345678}
{012345679}
...
{8}
{9}
{}
total 1024

Algorithm works, but clearly the implementation isn't reusable, because Backtrack algorithm uses hardcoded functions specific to the problem.

The simplest generalization that we can make is to make Backtrack function customizable - pass an interface that implements all specific methods to the Backtrack function!

Extracting required functionality to interface

Let's define our interface - it should have almost the same methods that has our concrete implementation:

type BacktrackerImpl interface {
    IsSolution(a []interface{}, k int, input interface{}) bool

    ConstructCandidates(a []interface{}, k int, n interface{}) []interface{}

    ProcessSolution(a []interface{}, k int, n interface{})
}

Notice that slice of bool []bool was replaced with slice of empty interfaces []interface{}.

Empty interface is a powerful tool - it describes a type that implements at least 0 methods, which means that every type satisfies empty interface. Thanks to this property we could use our Backtracker with bools, integers and even with Cats (defined as named structs).

Changing code to implement interface (close look at empty interfaces)

Now we have to modify our subset algorithm to implement this interface:

// SubsetImpl implements BacktrackerImpl interface.
type SubsetImpl struct {
    total    int
}

func (b *SubsetImpl) IsSolution(a []interface{}, k int, n interface{}) bool {
    return k == n
}

func (b *SubsetImpl) ConstructCandidates(a []interface{}, k int, n interface{}) []interface{} {
    // 1. We can't just return []bool{true, false} here.
    result := make([]interface{}, 2)
    result[0] = true
    result[1] = false
    return result
}

func (b *SubsetImpl) ProcessSolution(a []interface{}, k int, n interface{}) {
    fmt.Printf("{")
    for i, v := range a {
        if v.(bool) { // 2. We convert the type from interface{} to bool.
            fmt.Printf("%d", i)
        }
    }
    fmt.Printf("}\n")
    b.total++
}

First we create a struct with one field - total. total was previously a global variable, now it's stored inside the instance of the struct.

We then define methods required by the interface. They take as a receiver pointer to the instance of SubsetImpl type.

It shouldn't be a big problems, because we had those functions (ConstructCandidates, ProcessSolution, etc) defined in our first algorithm. The only difference is that those functions took slice of bools as a first argument and we have to change it to slice of empty interfaces.

To achieve this we only need to make two two small changes:

Instead of returning []bool{true, false} we need to allocate slice of empty interfaces and fill it with boolean values. Slice of empty interfaces isn't 'any slice'
We convert the type from interface{} to bool. If the value stored in empty interface was of some other type, program will panic. You can learn more about type conversions from type assertions and type switches article.

Changing original code to use new interface

Backtrack function already takes three parameters, so at this point it's better to make it a method of a type and store data in a struct, than add another parameters.

type Backtracker struct {
    impl BacktrackerImpl
}

// New creates new backtrackers, it takes implementation of backtracking
// problem as an argument.
func New(i BacktrackerImpl) *Backtracker {
    return &Backtracker{impl: i}
}

// Backtrack explores solutions of the problem. It's a little simplified
// version (working for all concrete examples in this blogpost).
func (b Backtracker) Backtrack(a []interface{}, k int, input interface{}) {
    if b.impl.IsSolution(a, k, input) {
        b.impl.ProcessSolution(a, k, input)
        return
    }
    k++

    candidates := b.impl.ConstructCandidates(a, k, input)
    for _, c := range candidates {
        a[k-1] = c
        b.Backtrack(a, k, input)
    }
}

Looks like we have all the pieces we need! Our new implementation can be used as follows:

func main() {
    a := make([]interface{}, 10)
    impl := &SubsetImpl{}
    b := backtrack.New(impl)
    b.Backtrack(a, 0, 10)
    fmt.Println("total", impl.total)
}

Using backtracking algorithm to solve another problem

Another implementation example in generator of permutations. It uses integers instead of bytes.

type PermutationImpl struct {
    total    int
}

func (b *PermutationImpl) IsSolution(a []interface{}, k int, n interface{}) bool {
    return k == n
}

func (b *PermutationImpl) ConstructCandidates(a []interface{}, k int, n interface{}) []interface{} {

    inPerm := make(map[int]bool)

    for i := 0; i < k-1; i++ {
        inPerm[a[i].(int)] = true
    }

    result := []interface{}{}

    for i := 1; i <= n.(int); i++ {
        if !inPerm[i] {
            result = append(result, i)
        }
    }

    return result
}

func (b *PermutationImpl) ProcessSolution(a []interface{}, k int, n interface{}) {
    fmt.Println(a)
    b.total++
}

Permutation implementation can be used in similar way to SubsetImpl:

impl := &PermutationImpl{}
b := backtrack.New(impl)

Thanks to empty interfaces everything just works!

Summary

I hope that you enjoyed this blogpost. Definitely reusing code in go can be challenging without inheritance and generics, but it's definitely possible.

This example doesn't exhaust all the possible ways to write reusable code in go. I would like to hear your battle stories from the work on go codebases, share your experience in the comment section or write me an email at justyna.ilczuk@gmail.com.

You can see all code from this blogpost in github repo.

API for resource with tags with django and DRF

Thu, 04 Jun 2015 00:00:00 +0100

Recently I was working on new exciting part of syncano-platform and I wanted to create a piece of api that would allow for filtering resources with many to many relationships.

I wanted to have a resource that can be tagged.

Adding such tags should be RESTful and straightforward. Also I wanted to add the ability to search using tags. My go to Rest library is Django Rest Framework.

Currently the most popular solution for tagging in django is django-taggit, however it doesn't integrate with Django Rest Framework well and heavily uses ContentTypes. In my current project at work we don't use ContentTypes at all.

It's also a pretty old library and has support for old django, south migrations, stuff that we just don't need in new projects.

I've read in a blog post about django-taggit:

Tags can be hard if you try to implement them yourself, so don't try and just use this library from the start.

And well, I disagree with this quote, how hard can it can be?

I sometimes prefer to write a simple code, well tailored to my use case, that install another library with all its dependencies. If library is poorly written you'll waste a lot of time debugging its strange behaviors and fighting with it when trying to tailor it your custom requirements. (I'm looking at you python social auth)

I decided to just do it myself. We will now distill this experience by building a simple app with two resources and REST API for them. Here is a code for a whole app.

App Ingredients

python 3.4
django 1.8
Django Rest Framework 3.1
django-filter
django-cors-headers 1.1.0

For simplicity I will use sqlite as a DB.

Data models

Let's imagine that we're building a very simplified version of PYPI. We're indexing libraries, we store their names, websites, descriptions and tags.

User can then search libraries using names, descriptions and tags, they can even add new tags to them.

How we will build it? We will have two resources

tags
libraries

In django, models are defined in models.py

models.py

from django.db import models


class Tag(models.Model):
    """Tag for data. Every tag has unique text.
    """
    text = models.CharField(max_length=64, unique=True)

    def __str__(self):
        return 'Tag[id: {id}, text: {text}]'.format(
            id=self.id, text=self.text)


class Library(models.Model):
    """Library represents a piece of software. It has its website url,
    name, description and as many tags as you like.
    """

    # django has a nice field that validates URLs
    url = models.URLField()

    # name should be in a slug form
    name = models.SlugField(max_length=80)
    description = models.CharField(max_length=256, blank=True)
    tags = models.ManyToManyField(Tag, related_name='libraries')

    def __str__(self):
        return 'Library[id: {id}, name: {name}]'.format(
            id=self.id, name=self.name)

And we will create nice RESTy api for them. It will be possible to add tags to libraries and vice versa.

To use django rest framework with those models, we need at three things:

serializers
views
routing

I usually start with serializers, serializer will translate our models to serialized date (it can be other way round too). I'm going to use simple serializers based on models directly. DRF has ModelSerializers for that.

serializers.py

from rest_framework import serializers

from .models import Library, Tag


class TagSerializer(serializers.ModelSerializer):
    libraries = serializers.SlugRelatedField(
        many=True,
        read_only=True,
        slug_field='name'
    )

    class Meta:
        model = Tag


class LibrarySerializer(serializers.ModelSerializer):
    tags = serializers.SlugRelatedField(
        many=True,
        queryset=Tag.objects.all(),
        slug_field='text'
    )

    class Meta:
        model = Library

The only tricky part there was to define SlugRelatedFields correctly. You can read more about related fields in DRF here.

With serializers in place, we can move to views part. Views let our app to communicate with the rest of the world.

In DRF you can either use views or viewsets. Views are simpler to understand, because they have the same methods as http verbs. Viewsets simplify lots of things and they are on higher abstraction level, so instead of post, you have create method.

Also viewsets combine very well with routers and usually using viewset results in using much less code.

Here I will use ModelViewSets, that gives us full CRUD functionality for free.

views.py

from rest_framework import viewsets

from .models import Library, Tag
from .serializers import LibrarySerializer, TagSerializer


class TagViewSet(viewsets.ModelViewSet):
    queryset = Tag.objects.all()
    serializer_class = TagSerializer


class LibraryViewSet(viewsets.ModelViewSet):
    queryset = Library.objects.all()
    serializer_class = LibrarySerializer

Simple, isn't it?

Now, let's connect views to urls.

urls.py

from rest_framework.routers import DefaultRouter

from .views import LibraryViewSet, TagViewSet

router = DefaultRouter()
router.register(r'libraries', LibraryViewSet)
router.register(r'tags',TagViewSet)
urlpatterns = router.urls

That's actually all we need to create DRF browsable API for those resources.

Adding filtering

We can add new things now, but finding something is currently a pain.

Let's implement filtering! We will enable filtering of libraries by:

tags
name
description

DRF has a pretty good support for filtering. You can read more about it here.

Even though support is good, we still have to do some things:

create a custom filter
enable filtering Backend

Let start from creating a custom filter. We will put in in separate file, so it will be easier to find.

filters.py

import django_filters

from .models import Library, Tag


class LibraryFilter(django_filters.FilterSet):
    # default for CharFilter is to have exact lookup_type
    name = django_filters.CharFilter(lookup_type='icontains')
    description = django_filters.CharFilter(lookup_type='icontains')

    # tricky part - how to filter by related field?
    # but not by its foreign key (default)
    # `to_field_name` is crucial here
    # `conjoined=True` makes that, the more tags, the more narrow the search
    tags = django_filters.ModelMultipleChoiceFilter(
        queryset=Tag.objects.all(),
        to_field_name='text',
        conjoined=True,
    )

    class Meta:
        model = Library
        fields = ['name', 'description', 'tags']

To create a custom filter, we had to inherit after django_filters.FilterSet, define our model in Meta and define how each of filters should behave.

django_filters offers few types of lookup_types for example: exact - that would match an exact string, 'contains', that would returns matches that contain given string, you can add i before those lookup_types, to make them ignore case.

The only tricky part is filtering by related field using its field, different than foreign key. I've seen some examples with filtering by related fields using foreign key, but I couldn't find in any docs, how to do it with different field (in our case - tag text).

Eventually I had to dig into source of django fields to find it out. It's enough to set to_field_name='field_name' and it should work.

To connect filter with view, you have to add following too lines to the view:

filter_backends = (filters.DjangoFilterBackend,)
filter_class = LibraryFilter

You can also set filtering backend in django app settings if you like.

Here is how it should look like in context:

views.py

from rest_framework import filters, viewsets

from .filters import LibraryFilter
from .models import Library, Tag
from .serializers import LibrarySerializer, TagSerializer


class TagViewSet(viewsets.ModelViewSet):
    queryset = Tag.objects.all()
    serializer_class = TagSerializer


class LibraryViewSet(viewsets.ModelViewSet):
    queryset = Library.objects.all()
    serializer_class = LibrarySerializer
    filter_backends = (filters.DjangoFilterBackend,)
    filter_class = LibraryFilter

Using our API

You can now list all tags:

$ curl http://127.0.0.1:8000/tags/

Here is how result will look like (if you've created some data before):

[
  {
   "text": "python",
    "libraries": [
      "django",
      "django-rest-framework"
    ],
    "id": 1
  },
  {
    "text": "web",
    "libraries": [
      "django",
    ],
    "id": 2
  }
]

If you curl libraries:

$ curl http://127.0.0.1:8000/libraries/

your result will look like this:

[
  {
    "description": "cool web framework",
    "name": "django",
    "url": "https://www.djangoproject.com/",
    "tags": [
      "web",
      "python"
    ],
    "id": 1
  },
  {
    "description": "api framework",
    "name": "django-rest-framework",
    "url": "http://www.django-rest-framework.org/",
    "tags": [
      "python"
    ],
    "id": 2
  }
]

You can now perform queries with filters like below!

$ curl http://127.0.0.1:8000/libraries/?name=rest&tags=python

[
  {
    "description": "api framework",
    "name": "django-rest-framework",
    "url": "http://www.django-rest-framework.org/",
    "tags": [
      "python"
    ],
    "id": 2
  }
]

Pretty neat!

Summary

I hope that you'll find this blog post useful. Building an APIs can actually be a pretty complex task, because there are so many components that have to be connected.

Also, we've just scratched a surface. Our API doesn't have proper authorization, paging, ordering, permission system or tests, we also have pretty scarce validation, no api versioning and we only handle JSON.

You can download or read full code on github.

Debugging - into rabbit hole with docker containers

Sat, 30 May 2015 00:00:00 +0100

This post will tell a story about successful debugging session. As an author of a portion of code, we are naturally biased to not to think about worst ways of exploiting it. Luckily I have creative colleagues. Recently one of my colleagues did some additional testing and left me with a destructive, yet interesting bug related to Docker, sandboxing and abuse. In this post I will describe how I approached and fixed it.

In syncano - company I work at - we allow for custom code execution, it’s a part of our Backend as a Service solution. We run our code in Docker containers which should be more less safe if enough security is in place. We do such things like executing this code on separate machines that rest of the tasks. Those machines have less privileges and less knowledge about the system and containers itself are constrained on things like memory usage and time of execution.

So I was really surprised recently that snippet such as:

while True:
    print(“woopdido!”)

could cause us serious trouble. The symptoms were that codebox (abstraction for wrapper around container and code) wouldn’t time out, but will stay in the pending state. We would expect it to be terminated after 30 seconds. Moreover codeboxes scheduled next, after the one that was stuck, would also stay in pending state.

In our monitoring platform we could see that load on this kind of workers was surprisingly high and that codeboxes that were meant to time out after 15 or 30 seconds could run for hours! Clearly it was a bug in our implementation of timeout subroutine.

It was a bit surprising, because I remembered implementing this part of code and writing tests for it. If there was a regression in this part of code, we should catch it on our Continuous Integration server during build. Tests were green.

I’ve remembered that in our integration test cases I used time.sleep and quickly tested a codebox with sleep on our staging:

import time

while True:
    time.sleep(0.1)
    print(“woopdido!”)

And this time, the behavior was correct. Maybe it was connected to some kind of waiting? We used to send a signal to container to shut it down if hasn’t finished on time.

I quickly verified this hypothesis by trying out another example:

x = 0
while True:
    x += 1
return x

This time behavior was also correct. So it wasn’t specific for using time! It was pretty late when I found it out, so I just wrote a small note in the bug ticket. Next step was to recreate the same problem on my local machine next day.

Recreating the bug was quite messy. One side effect was making my machine swap. One of the first things I noticed was that we were using the pretty old version of the docker library for python.

I freezed version 0.6 and there was version 1.2.2 out there when I started debugging it. Time passes really fast. From one side of view, it’s bad that we haven’t updated this dependency for half a year, from the other side - it’s nice to have a piece of code that doesn’t have to be modified often.

Anyway, it turned out that new version has timeout functionality built in. Great, I thought, probably my custom implementation of timeout was wrong in some way and the official one would handle my case better.

Actually - it didn’t change anything. I also noticed during testing with few flavours of problematic scripts, that the whole problem it’s not really about timeout - but about resource usage. Even if container was killed on time, memory wasn’t returned to the OS. So problem lied somewhere else. My next suspect was extremely verbose printing.

I measured how much logs can be written to a file during 15 seconds assuming simple printing. Not too much - it would take long to display with tool such as less, less than few hundreds Mbs. It definitely didn’t match the behavior of eating all my RAM (16 Gb on my private laptop and 8Gb on my work machine). However if I did more ‘printing’ problem only had gotten worse:

for _ in range(1000000):
    print(“hello” * 1000000)

Would it my RAM with speed of half Gb per second - and before actual time out machine would start swapping like crazy. It shouldn’t happen, because newly spawned containers were memory constrained and not even one of my containers seemed to be busy - by observing their processes I would rather say that they were bored. Problem was probably on host.

I could easily observe the RAM / cpu disaster in a window with htop opened. There were two ways to get out from this situation - restart my computer or restart docker. Both very tedious, because with whole environment using docker it meant restarting the whole stack every time.

When I tried to access logs from the container which was running codebox containers, for few times I got an interesting exception:

# docker run -ti ubuntu:trusty bash
FATA[0028] Error response from daemon: Cannot start container 02533d9e9afcc81818d3d71aa7498e0f41e7904483fd041fa45a0638423a5d3a: [8] System error: fork/exec /usr/bin/docker: cannot allocate memory

Initially I thought that it was because docker had eaten all my RAM, but it turned out that I could do other things that required memory allocations. Puzzled, I googled this exact exception message.

It led me to the issue on docker’s github page. The main asker describe the similar problem, verbose logging resulted in big memory leaks. Bingo! That looked roughly like my problem.

github issue

memory leak in buffer grow #9139 Closed e-max opened this issue on Nov 13, 2014 · 29 comments

Memory leak issue in buffer grow. It sounded nasty, but I decided to read all the comments. Some devs suggested using different logging handlers available in new versions of docker, but it didn’t work for me - I had to restart my environment few times more. Moreover on the end of the issue page Docker devs say that those problems won’t be fixed until 1.7 (probably).

Sometimes you’re left to your own devices. I thought for a while about creating a patch for Docker, but it wouldn’t be there until the next release, I also wouldn’t be able to use my own patched version of docker, because we have some other environment constraints. Well, I’m also a very novice golang programmer and I had no experience with fixing memory allocation problems. So I had to think about workaround on our side.

If you’ve spend some time in terminal window on Linux or Mac, you’re probably familiar with stream redirection. If you can’t print to foreground, why not redirect it to the files? It’s a good solution because it’s language independent, so I wouldn’t have to worry about capturing streams in every language that we support for codebox. > sign in bash is much more elegant.

It sounded like the exact thing that I needed. I quickly changed the command string that was passed to newly created containers:

'python my_script.py' -> 'python my_script.py > logs'

I started the adversary container and… nothing changed. And my computer started swapping again. Duh, what could go wrong?

I remembered that sometimes writes in Copy On Write filesystems can give worse performance, than those on more typical ones, such as EXT4, however this behavior was clearly strange. I tried to mount logs file as volume, so that I would omit COW filesystem.

It didn’t work, not only my computer also ended up without free RAM, I also couldn’t see any logs in the mounted files. It was strange and on this point I was also pretty tired. I got some coffee, juggled for five minutes and helped fellow devops in his problem.

I clearly wasn’t redirecting correctly to the files. Luckily, the offensive containers were running for 15 to 30 seconds depending on my timeout configuration and I could make sure that stdout was redirected by looking up container’s logs.

$ docker log -f nifty_ritchie
logging
logging
logging
….

They weren’t redirected correctly. By sheer accident I remembered that I’ve read somewhere on some blog or even in docs, that wrapping calls of docker commands in bash -c ‘original bash stuff’ could solve some unexpected behavior.

That was it.

 'bash -c ‘python my_script.py > logs 2> errors’'

was a winning combination.

Summary

I was lucky that it took me only one day to debug this problem. Bugs always seem obvious in retrospect, but can be a real rabbit hole and consume lots of your precious time that could be spent on developing features.

Here are my rules of thumb for debugging:

recreate your bug in local environment that you control (if that’s possible)
write your hypothesis down
write down your assumptions and try to test them as fast as possible, preferably in isolation
always try to learn more about the problem
take breaks - thinking in front of computer can be really hard, especially if you’re deeply involved into problem, it's also very important to avoid getting unproductive and wasting your time, I've recently seen a very good talk 'Into the Rabbit Hole' by @asendecka that talks about exactly that.
talk to people :) - if they’re technical, they will help you spot flaws in your thinking, if they aren’t technical, at least you’ll get some perspective during the break

Debugging is more like art than a science. I would like to hear about your opinions and your best techniques. Just comment below, talk to me on twitter on @attilczuk or write me an email to justyna.ilczuk@gmail.com.

Docker Meetup in Warsaw

Sun, 12 Apr 2015 00:00:00 +0100

You've probably heard about Docker already, it's a hot DevOps technology that gained popularity during last two years. Last month I've organized a first Docker Meetup in 2015 in Warsaw.

I did it with a help of my great team at Syncano, the company provided office, food & drinks and all needed infrastructure.

The website for the meetup is here. Almost 60 people made RSVPs (we had to limit number of people, because we were little worried if our office can handle it).

We had two talks:

Docker in Everyday Development - 30 min, Justyna Ilczuk from Syncano (me ;) )
Introduction To Kubernetes - 45 min, Filip Grzadkawski from Google

and workshops during which we were practicing working with Docker CLI and installed docker-compose (which wasn't so easy on fedora! It started working after doing it from the virtualenv).

Workshops were mostly improvised, but turned out to be well received. I really enjoyed working with Tomasz Cholewa (coorganizer of the workshops). I have a development background and Tomek is an Ops person - combining our perspectives was an interesting experience. We've met for the first time on this workshops.

Apart from workshops, I prepared a talk about using Docker in everyday Development. I talk there a bit about reasons for doing it and then I move to tools that I recommend for creating development environments with Docker.

If you like my talk, pass it on. If you have some questions, ask them here or drop me an email.

Organizing this meetup was a great experience and I don't want to stop on one event, so if you have some interesting talk that you would like to give in Warsaw next month, contact me.

Bash monitoring tips - watch & tee

Sun, 22 Mar 2015 00:00:00 +0000

Recently I was debugging some problem with deployment in Syncano (where I work) and I wanted to catch an exact moment when the problem occurred. Docker was dying on a host without any good reason.

Let's assume that I wanted to observe what Docker containers are running at a given moment.

Here is my snippet, we will later describe it in detail:

$ watch -n 3 'date >> dump_ps && docker ps | tee -a dump_ps'

It uses:

watch
date
docker ps (but can be replaced by any command that you're interested in)
tee
tmux (later in the blog post)

What it does?

It executes a command in single quotes 'date >> dump_ps && docker ps | tee -a dump_ps' every three seconds, displays it on the screen and appends to file.

The command consists of two parts date >> dump_ps and docker ps | tee -a dump_ps, both of them append to dump_ps file.

date >> ps is pretty self-explanatory, because it just appends a current date to a file. You can pass arguments to the date program to adjust the format - date man page.

If we typed:

$ watch -n 3 'date >> time_is_passing'

and interrupted after few seconds, we could see the time written to time_is_passing file:

$ cat time_is_passing 
nie, 22 mar 2015, 03:57:08 PDT
nie, 22 mar 2015, 03:57:11 PDT
nie, 22 mar 2015, 03:57:14 PDT
nie, 22 mar 2015, 03:57:17 PDT

Output may differ if you have different locale / timezone settings.

The second part of the command: docker ps | tee -a dump_ps is a little more complicated.

If you have docker installed, docker ps will show you info about the containers running on your host. More info about Docker command line commands may be found here.

If you don't want to play with docker, you can replace this command with top - it will show you your processes instead.

Ok, so the first command would normally display an output on the screen or pass it through the pipe to another program. What we want to do here is to both show and append the text to the file. There is a program just for that in every Linux distro. It's called tee.

Image from wikipedia article about tee.

$ echo "I love command line" | tee interests
I love command line

$ cat interests 
I love command line

However if we wrote another interest to the file in a similar way:

$ echo "I love climbing" | tee interests
I love climbing

$ cat interests                         
I love climbing

We would replace the whole file! It's because tee without arguments is like using >. To append to a file pass the -a flag to tee.

$ echo "I love command line" | tee -a interests
I love command line

$ echo "I love Linux" | tee -a interests
I love Linux

$ cat interests                         
I love climbing
I love command line
I love Linux

Cool! We can now build an ad hoc dashboard that refreshes every three seconds and dumps all the info to the file.

But what if we wanted to monitor more than one thing at once on the same screen? Let's enter the world of tmux.

Tmux is a terminal multiplexer - if you haven't heard of it, grab a tutorial.

It can be installed with apt-get or yum:

sudo apt-get install tmux

sudo yum install tmux

Ok, so if you have tmux installed, type:

$ tmux
$ Ctrl-B+"

Your screen should split vertically. Then run the command in the first pane and switch to the other one with Ctrl-B+Up Arrow and run the next command.

Here is a result from my laptop:

I hope that you've learned something new. Shell is full of small gems that really shine when used together.

Happy hacking!

Maker Productivity 101

Sun, 30 Nov 2014 00:00:00 +0000

In this blog post I want to share with you some of my best reflections about how to be productive with many serious, projects at the same time.

Maker schedule vs Manager schedule

Paul Graham differentiates in his article between two schedules:

maker
manager

In manager schedule - you change what you're doing every hour - meetings, calls, urgent events. It's about managing resources and reacting to changes.

On the other hand in maker schedule you work in bigger chunks of time - such us a block of five hours - to focus on complex things, research, implement and create. It's often accompanied with a state of flow.

GTD and todo lists

If you're interested in productivity - you've probably heard already about GTD philosophy - write down, manage, set priorities, execute tasks from the lists you produced (it's actually much more complex system).

However not everything reduces to clear and easy to accomplish next actions. Sometimes there is also a [deep work] to do.

And to do something really meaningful we don't need a better list of next actions, but we need to train our ability to focus on meaningful things for long periods of time. GTD works for the schedule of manager, but for makers (researchers, artists) it really doesn't make sense.

There are also TODO lists, which are very helpful when your life is in a complete chaos and you want to write things down to get it out of your head and see what's really there to prioritize. However maintaining TODO lists for long periods of time can be bad for you - mostly because tasks tend to accumulate and long lists can fill your free time with lots of triva, not mentioning making you feel overwhelmed or guilty that you have so many tasks to do.

Another problem with small tasks I see is that they demand frequesnt context switching. If I have 4 big projects and I chunk the time for them to 30 minutes blocks - it turns out awful - I don't get things done - even though I spend this time, also - I don't enter flow state which I'll describe later in the article.

My another big problem is that I cannot stop thinking about next things after the chunk and I start micromanaging myself - which turns out to be stressful.

Flow

Flow is a very productive and focused state of mind, person experiencing flow can be totally immersed in the task. You can read more about it's definition here.

In short the biggest benefit of flow is that you can get a lot done and get lots of satisfaction from it, because flow just is an immensely rewarding and fulfilling experience.

However to get into the state of flow you need:

challenging task with clear goal
enough uninterrupted time to gain focus
freedom of distractions
enough skill, so that the task won't be impossible for you

Flow is one of the most important reasons why I love programming - I often catch myself totally immersed into programming task.

For long time I didn't know a lot of people working in this way. When I first met some 'makers' they looked totally disorganized, yet they turned out to be surprisingly productive. It changed the way I approach productivity now.

Focus

I'm currently pursuing more projects that I think it's sane to pursue in the same time. They are:

working almost full time as software engineer in a really exciting startup
finishing my degree in applied physics - I attend some classes, have some projects to do and tests
writing a bachelor's thesis (complex systems, statistical nonequilibrium physics, math - hypergraphs)
writing a technical book for Packt Publishing - writing on this blog
constantly improving my skills as software engineer - learning tools and frameworks, learning fundamentals, security, devops, operating systems and more
building things in local hackerspace - lately quadcopters
some minor personal projects

I would also want to contribute more to open source, read more books and exercise more - however the last one isn't so bad actually, because I manage to work out at least once a week.

However - it's easy to spot problems with this list - it's too long. Apart from university and work eating up at least 10 hours a day of my time, I have approximately 6 or maybe even 4 hours (social life, relaxing?) I could do some work. As I mentioned previously half an hour blocks don't work for me.

How to gain focus then?

Remove distractions

My biggest distraction is most of the time is browsing the internet.

Turn off the internet connection in your computer. Print what you want to read. Work in a room without your computer, use paper!

Or if you have to work with computer, turn off all the unnecessary tabs and communicators, to leave only what you need to do your work.

Socialize less

Yes, it might be a bit strange to say this - because meeting new people and spending time with friends is usually considered a good thing in life. But when your time is sparse you have to make space for quiet time just for yourself to push things forward.

Choose one thing or max two

Having 6 ongoing projects doesn't mean that you have to do progress everyday in all of them.

Instead of doing really small chunks of easy work in all of them, choose one project and make a big progress in it.

Store possible tasks and remove them

There are lot's of cool articles in the internet - both technical (potentially useful) and sociopolitical - which may grab your attention. Reading an article can take approximately 10 minuters, it's not really so much right? However it usually doesn't stop on one, but more likely on 10 articles in the row - just before you wanted to do something, or just when you took a small break. And 100 minutes is can make a real difference I you spend it on meaningful projects.

How to resist the urge of reading the article now - in this particular moment? My answer is - save it for later.

I discovered a Pocket. It's a simple app for saving what you want to read later and synchronizing it across all of your devices so you can read it offline on your phone or using different computer.

It also has one killer feature - it's possible to remove items from the list - and your list should be short.

Other tips

Books over articles

Although articles are nice to get started in some areas, it's often much better to read a proper book on a subject. Reading a technical book 100 minutes per day will greatly improve your skills and knowledge in contrast to articles on the web.

What books are you reading right now? Do you feel that you can't get through them or there are just to many books you want to read?

Or maybe you aren't reading anything from your field right now? Why?

Track how much time you spend on your projects

It might turn out that you're not productive in some of your pursuits, because you don't spend enough time on it.

The best way I found so far to motivate myself to spend time on something and accurately track it in the same time is beeminder. In beeminder you set your goals - like doing more of something, and you declare your commitment weekly. If you don't meet your goals, you pay money to beeminder and amount of money at stake increases.

It also gives you insights how big is typical daily chunk you invest into doing something. I love it.

Summary

There is no silver bullet - however focusing and prioritizing helps a lot. I hope that my tips and tool would be useful for you. Thanks for reading and get back to work ;).

Virtualenv the Docker way

Sun, 16 Nov 2014 00:00:00 +0000

This post will describe how to use virtualenv's tricks to run python inside Docker container extending concept of python virtual environment, without changing any current habits.

If you develop in python you've probably heard about pip and virtualenv, if not... go read about them now. They're pretty standard way of managing dependencies in python.

Virtualenv & pip workflow

Typical workflow with virtualenv and pip looks as follows:

$ virtualenv .
$ source /bin/activate

In the above steps we create a virtual environment and activate it.

(docker-virtualenv)
# att at lapunov in ~/Projects/docker-virtualenv on git:master x [8:05:05]

With virtualenv we can start using pip without worrying that it will pollute our global space.

$ pip install -r requirements.txt
$ pip install my_new_dependency

When we no more want to use a virtualenv, we can get rid of it by:

$ deactivate

Simple, right?

All dependencies are in docker now...

Pip and virtualenv work well and you get used to them, but here comes Docker and lots of things change. Now, you install all dependencies inside a container instead of virtualenv

Installing dependencies in docker has it's advantages over using just virtualenv - because thanks to containers - you don't have to worry about system's dependencies, and your app is now better isolated.

However running python becomes a bit cumbersome, because you don't have dependencies installed on your machine, and you would have to get into docker container to run python there.

For example you work with Django and want to run ./manage.py shell_plus, as you always did, in a current directory?

Or run ./manage.py collect_static, or just play around with packages you've just installed using IPython?

You first have to enter a docker container. And depending on your Dockerfile you can have few things wrong - default directory, user, environment variables...

It isn't the nicest developer experience ever. The best thing would be to not change our habits at all.

Virtualenv's `bin/activate`

What if we have something like virtualenv which would change our python so it runs inside the Docker?

All we have to do this is to change how python is executed.

Instead of running a python binary we can run a python bash script, which will run python inside a Docker container containing all our installed dependencies.

Do you remember what virtualenv does? It creates a bunch of directories:

bin  lib  man  share

And stores a new python, pip and so on in bin directory as well as magical activate file which shouldn't be run but sourced.

When we do:

$ source bin/activate

We change python to the local one. activate is quite simple and it uses the fact that python looks for python executable on using the PATH environment variable, and uses the first python executable it finds. So activate prepends virtualenv bin path to $PATH.

$ echo $PATH         
/home/att/Projects/docker-virtualenv/bin:/usr/local/sbin:/usr/local/bin
(docker-virtualenv)
$ echo $VIRTUAL_ENV 
/home/att/Projects/docker-virtualenv

Fake your python

We can do a similar thing as virtualenv does. Let's create a bin directory:

$ mkdir bin
$ cp ../my_other_project_with_virtualenv/bin/activate bin/activate

and change: VIRTUAL_ENV environment variable to pwd:

VIRTUAL_ENV=`pwd`

Now we only have to create a fake python executable.

$ echo "echo 'hello'" > bin/python  # create a dummy python
$ chmod +x bin/python  # make script executable

So we should have now a structure like this:

$ tree bin
bin
├── activate
└── python

To use our new python ;), we have to source activate again.

$ source bin/activate
$ python             
hello

Hurray! It means that we have dummy python in place, and it's a default python now. We only have to make it a little bit more useful and run a python inside a Docker container.

This is an example content of bin/python:

$ cat bin/python
#!/bin/bash
docker run -it docker_image_name python "$@"

If you rewrite a bin/python to the version above and reload activate, you run python in Docker just like always:

$ python

But it will be little slower to start...

$ time python -c "print('hello')"
hello
python -c "print('hello')"  0,02s user 0,01s system 10% cpu 0,318 total

However - it works!

If you work with fig - tool for isolated Docker development environments, you probably should use a fig run instead of docker run - because it will setup also a rest of containers for you with proper links, volumes and environment variables. Running with root privileges without a good reason is a bad practice so it should be avoided. I use a docker/baseimage as base for some of my Docker images, so I have a /sbin/setuser command which I use to run commands as a different user.

After the changes mentioned above (assuming using fig and baseimage) bin/python might look as follows:

#!/bin/bash
cd /home/att/Projects/docker-virtualenv  # fig has to find `fig.yml`
fig run docker_image /sbin/setuser virtualenv python "$@"

Now you don't have to remember that your python app is in Docker. Just use python as you always do :).

And to restore normal python, just run

$ deactivate

Side effects of mock

Sat, 08 Nov 2014 00:00:00 +0000

What is mock

Python mock module allows you to replace parts of your system under test with mock objects and make assertions about how they have been used.

Mock is available in the standard library in python3 as part of unittest module. You can read it's documentation here. In python 2, mock is available to install with pip.

Use cases

What do we use mocks for? Mock are mostly used in unit tests. We try to make tests as short and focused on what we test as possible.

Mock call to external service

As I stated in the first paragraph, mock is mainly used for testing. Recently I was implementing a social signup and I didn't want to call facebook every time, but I wanted to test if the signup works as it's supposed to.

Calling an external service such as facebook is really expensive, because http requests take time (hundreds of milliseconds at least) and requires thinking about lots of complex things - such as if your access token is still valid. And we usually don't want to think about it during testing, because we want to test this one specific detail of our code.

Below is a short gist of this usage of mock:

get_json_mock = mock.Mock(return_value={'name': 'John',
                                        'email': 'john@doe.com'})

class SocialAuthViewTest(APITestCase):
    backend = "facebook"

    def setUp(self):
        test_token = 'token test_facebook'
        self.url = reverse('authenticate_social', args=(self.backend,))
        self.client.defaults['HTTP_X_AUTHORIZATION'] = test_token

    @mock.patch('social.backends.base.BaseAuth.get_json', get_json_mock)
    def test_returns_200_when_user_found(self):
        response = self.client.post(self.url)
        self.assertEqual(response.status_code, 200)

If we know which function is called and what it should return, we can patch it using @mock.patch decorator and previously created Mock object with return_value set to what we want to get as result of function call.

Locally turn off logging

Let's imagine that we've already implemented most of our social login functionality and now we want to test if our API properly reacts to bad logging in attacks.

We don't want user to see ugly 500 - internal server error HTTP error code, but 401 unauthorized HTTP error code.

When we get information from an external service that credentials were invalid we should log the accident. In tests our logger prints to standard out. But do we want an ugly traceback in our tests output? I would rather see something like this:

....................................
----------------------------------------------------------------------
Ran 17 tests in 1.610s

OK

Short and clean. But how should I keep loggers quiet?

I quickly searched the web to find good solutions to this problem. Stack overflow provides some answers but they weren't satisfactory to me, because most of them included changing logging too much, by changing it's level globally in tests.

I wanted a more fine grained approach and I used mock library for it.

First I had to find the the file logging the exception and understand how it's done.

Here is a file users/views.py

logger = logging.getLogger(__name__)

def sign_in_or_signup(register, access_token):
    try:
        return request.backend.do_auth(access_token)
    except requests.HTTPError:
        logger.exception('Invalid login')

It's a pretty standard django way to use logging. We create logger instance for the module and use it to print out messages to standard out or send details of errors to some external service such as sentry.

If we only could mock this logger to swallow exceptions...

users/tests/test_social_signup.py

    @mock.patch('users.views.logger', mock.Mock())
    def test_user_authenticates_wrong_access_token(self):
        self.client.defaults['HTTP_X_AUTHORIZATION'] = 'token iamjustwrong'
        response = self.client.post(self.url)
        self.assertEquals(response.status_code, 401)

Voilà, done! It's even easier than the previous example.

Testing a behavior flow

Another interesting usage of mock can be while testing check and retry if wrong pattern.

My coworker was testing this kind of behavior and used code similar to the one below:

def generate_value_mock(*args, **kwargs):
    iterator = getattr(generate_value_mock, 'iterator', 0)
    setattr(generate_value_mock, 'iterator', iterator + 1)

    if iterator < 2:
        return 'a'
    return 'b'

Nice piece of advanced python, right? Actually it demands thinking, it's complex. Maybe it even should be tested to be sure that it's behavior is correct. One of the testing rules is that we don't test tests, so they should be as simple as possible.

Guess what, we can use mock library to simplify this piece of code. Below is the gist from IPython representing how it works:

m = mock.Mock()
In [17]: m.test.side_effect = ['a', 'a', 'b']

In [18]: m.test()
Out[18]: 'a'

In [19]: m.test()
Out[19]: 'a'

In [20]: m.test()
Out[20]: 'b'

We can set side_effect on Mock object. Depending on what type of object we pass to it, we can get different behaviors when calling mock object.

Useful for raising exceptions or dynamically changing return values. The function is called with the same arguments as the mock, and unless it returns DEFAULT, the return value of this function is used as the return value. Alternatively side_effect can be an exception class or instance. In this case the exception will be raised when the mock is called. If side_effect is an iterable then each call to the mock will return the next value from the iterable. A side_effect can be cleared by setting it to None.

You can read more about this particular property here.

It's only a tip of the iceberg...

If you haven't heard about mock before, you might find this examples impressive, however they don't show even 10% of real capabilities of python mock library.

I recommending you reading it's detailed documentation or if you prefer more hands on approach, you can check out mock tutorial form Toptal.

Anyway, happy mocking!

Tiny struggles - engineering blog

Notifications on the channels in Golang

Introduction to the pattern

Ordering trap

Real life example

Golang Concurrency Exercises

Exercise 3 - vending machine

Exercise 4 - design simple work pipeline

Exercise 5 - control access to 3D printers

Exercise 6 - hackerspace membership fees

Summary

Golang Code Reuse - Generalization tricks

How to reuse code in Go?

Algorithmic example

Extracting required functionality to interface

Changing code to implement interface (close look at empty interfaces)

Changing original code to use new interface

Using backtracking algorithm to solve another problem

Summary

API for resource with tags with django and DRF

App Ingredients

Data models

Adding filtering

Using our API

Summary

Debugging - into rabbit hole with docker containers

Summary

Docker Meetup in Warsaw

Bash monitoring tips - watch & tee

Maker Productivity 101

Maker schedule vs Manager schedule

GTD and todo lists

Flow

Focus

How to gain focus then?

Remove distractions

Socialize less

Choose one thing or max two

Store possible tasks and remove them

Other tips

Books over articles

Track how much time you spend on your projects

Summary

Virtualenv the Docker way

Virtualenv & pip workflow

All dependencies are in docker now...

Virtualenv's bin/activate

Fake your python

Side effects of mock

What is mock

Use cases

Mock call to external service

Locally turn off logging

Testing a behavior flow

It's only a tip of the iceberg...

Virtualenv's `bin/activate`