mercuriete's blog

Keep Gentoo sane and not becoming insane

Sun, 10 Mar 2019 12:00:00 +0000

Keeping updated a Gentoo instalation is a task quite a few difficult. In this post we learn how to keep up with the updates.

TL;DR

#!/bin/bash
eix-sync
emerge --update --deep --changed-use --newuse --with-bdeps=y @world --ask
emerge @preserved-rebuild
emerge --depclean --ask
revdep-rebuild
revdep-rebuild.sh -u --pretend

See Gist

Eix-sync

eix-sync is a amazing tool that allows us to download the new state of portage. Doing an analogy to debian based distros is similar to apt-get update.

In addition, checks the differences between the last time you launch that command and show you problems with your installed packages.

* Calling eix-diff
[U]  ==  dev-libs/openssl  (1.0.2q@03/03/19;  0.9.8z_p8-r1(0.9.8)^d  1.0.2q^d  ->  0.9.8z_p8-r1(0.9.8)^d  1.0.2r^d): full-strength general purpose cryptography li
brary (including SSL and TLS)
[>]  ==  sys-power/bbswitch  (0.8-r1  ->  0.8-r2): Toggle discrete NVIDIA Optimus graphics card
* Time statistics:
26 seconds for syncing
7 seconds for eix-update
1 seconds for eix-diff
34 seconds total

For example, in this time, we got updates for the package called openssl that is already installed and bbswitch that is not installed in our system.

There are several codes for package status:

[U]: the package is installed and there is a new update available.
[>]: there are new updates for a package that is not installed (useless information but nice to know it).
[<]: Gentoo stop supporting a package and is not in portage anymore. You can keep your package installed but probably in a few month a dependency problem will appear and you will be forced to search an alternative.
[?]: There is something wrong with your package. Usully this means you have an unstable package installed and is only unmasked for a given version. The system is telling you that you need to update by hand this package unmasking a new version of this package.
[UD]: You had a unstable version installed but you forgot to update and now there is a new “old” version stable and the system is forcing you to downgrade. The solution is downgrade or unmask a new version.

Is very important to read carefully the information given by this command because if you ignore it, you will have problems in the future.

Emerge –update –wtf_are_those_arguments

This command is already in the official documentation: See documentation

–update: we are going to update packages
–deep: we are searching through the dependency tree in DFS (deep search) looking for dependency updates
–changed-use: we will rebuild a package that is not updated but you changed the USE flags that are affecting to this package.
–newuse: we will rebuild a package that is not updated but someone (usually the package mantainer) adds a new USE flag.
–with-bdeps=y: we will be updating the build deps and not only the runtime deps. For example, gcc, g++, bison, flex, are usually build dependencies but not used in runtime. Keep your toolchain updated is very important for new optimizations and stability.
@world: This is the package set you want to update, you can put here a single package or a set. World package means “all installed packages by me”: See documentation
–ask: Stupid mode. The command treats you as a stupid and ask everything. This argument is very important because It resolves unmasking problems such as a package is requesting an unstable package or you are trying to install a privative package without accepting the EULA (for example chrome-binary-plugins if you want see netflix)

emerge @preserved-rebuild

Sometimes, hardly ever, you will encounter a weird problem.

You are updating a given library but some other package depends on this, the system detects that there are some linking incompatibility and the system keeps two copies of the library for avoid break the application.

What emerge preserved-rebuild does is rebuild all applications that depends on this library in order to be able to delete the “preserved library”.

This not happens very often because emerge rebuilds applications automatically, this only happens when the emerge process is stopped badly and you don’t gave It the opportunity to keep the system in a sane state.

emerge –depclean –ask

This deletes orphan packages (packages installed that nobody want them).

If you are a debian based user, this is similar to apt-get autoremove. –ask argument stops the execution and prompt you a Yes/No confirmation, useful to avoid delete packages that you want to keep but you forgot to rebuild the application with that version of the library. This usually happens when you update llvm but you forgot to rebuild mesa.

revdep-rebuild

This command is very useful to keep your applications in a sane state.

What this command does is keep the links working properly. See documentation

Iterate over all applications
check the links that points to shared libraries
If the link is broken check to what package belongs that file
Installs or rebuild the library package

This situation is not very common but when It happen is difficult to track down.

revdep-rebuild -u –pretend

This command is very important, It will save you hours and hours of debugging and frustration.

What this command does is:

Iterate over all applications
Check all dinamic libraries and symbols that are exported from the library and imported to the application required to work. like: readelf -d $executable | grep 'NEEDED'
Figure out what package contains that library. like: equery belongs $file
Suggest you to rebuild that package.

This problem should never appears but the reality is that you can have problems with the hard drive (corrupted files), or you can have a big update of gcc.

When this events happens, you think your dependencies are correct but in reality the symbols that exports the library are different breaking the ABI and the application got broken.

With this advise you can avoid a full system recompilation that is suggested in official documentation: See documentation

Conclusions

The Gentoo official documentation is great but is splited in several locations so you have to learn which combination of commands are suitable for you.

In addition, keeping up with the updates without breaking the system is difficult so this is why Gentoo is not the best distribution for newbies.

Given that, the decision of moving from a given gcc version to another is something that someone have to do. I mean wheather you are building your packages or a package mantainer are building this for you, ABI changes are problem that someone have to solve. The only difference between Ubuntu and Gentoo is that those problems are resolved by other people that are not you, but the problems is still there.

With Gentoo you will learn more about linking problems, ABI breakages and some other problems that you are not facing in a binary distribution like Ubuntu or Windows or MacOSX

I Hope you enjoy this post.

Functional Programming with C++ 2017

Sat, 01 Sep 2018 12:00:00 +0000

Designing high performance applications is difficult without the knowledge of parallel programing. We will achieve the full usage of all cpu cores using functional programming and C++ 2017 parallel algorithms.

TL;DR

Parallel Euclidean distance:

$\sqrt {\sum _{i=1}^{n}(x_{i}-y_{i})^{2}}$

   vector<double> v1 = { 1.0, 0.0, 0.0 };
   vector<double> v2 = { 1.0, 1.0, 0.0 };
   //euclidean distance
   auto euclidean = sqrt(transform_reduce(execution::par,v1.begin(),v1.end(),v2.begin(),0.0,
                      [] (auto x, auto y) -> auto {return x+y;},
                      [] (auto x, auto y) -> auto {return (x-y)*(x-y);}
                     ));
   cout << euclidean << endl;

result:

DISCLAIMER

C++ 17 Standards are too new to be production ready. There are no big compiler vendors that provide STL parallel algorithms at this moment. All this examples are created using Intel® Parallel STL.

Status of gcc C++ 2017 Support

Status of MSVC++ 2017 Support

I will do a small tutorial for set up the environment but is only for linux + g++ + gentoo. If you don’t have this environment, you have to do by your own.

Intel® Parallel STL Installation

See documentation

Install gcc with support for OpenMP and C++11

sudo USE="openmp" emerge sys-devel/gcc --ask

Install Intel® TBB 2018

sudo emerge =dev-cpp/tbb-2018.20180312 --ask

Clone the Intel® Parallel STL

git clone https://github.com/intel/parallelstl.git

Remember the path of include folder for later

Functional Programming

Functional programming is a programming paradigm that allows the programmer to relax the definition of the control flow relying in our compiler .

When using imperative programming we are telling to the compiler exactly what It have to do, ending in a not too much performant code.

Example for loop

Imperative
```
for(auto i = 0; i<N; i++){
 something(array[i]);
}
```
You as a programmer are telling to the compiler to do the following:
1. Initialize iterator
2. do something with an array
3. Increment iterator
4. Test condition and jump backwards

As you can see this definition is too strict and the compiler can’t optimize your code because It can’t understand what you want to do.

Functional Programming
```
for_each(array.begin(), array.end(),
[] (auto element) -> auto {return something(element););
```
As you can see the meaning of this loop has changed a little bit to the following:
1. Do some stuff with each element. I don’t care how you are doing It. But I still want to preserve the execution order.

C++ 2017 Parallel Algorithms

Once we understand the power of functional programming and the compiler have more room to improve our code, C++17 defines a way to run code very efficiently.

Execution Policies

See documentation A execution policy is a way to hint the compiler If you want to continue executing your code sequentially or if you want the compiler parallelize for you.

std::execution::seq: I want to preserve the original sequential behaviour.
std::execution::par: I want the compiler to parallelize my code
std::execution::par_unseq: The compiler tries to parallelize and vectorize the loop.

Example of Parallel STL Code

Understanding the STL Algorithms is not in the scope of this tutorial. It is granted you are a c++ developer and know some of the following algorithms:

#include <iostream>
#include <vector>
#include <cmath>
#include "pstl/algorithm"
#include "pstl/execution"
#include "pstl/numeric"
using namespace std;

const int MAX_SIZE = 100000000;
int main(){
    //initialize vector with zeroes
   vector<int> v = vector(MAX_SIZE,0);
   //initialize vector with ones
   transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return 1;});

   //get some random numbers
   transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return rand()-(RAND_MAX/2);});

   //calculate ABS
   transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return abs(x);});

   for_each(execution::seq,v.begin(), v.end(), [] (int x) -> auto { cout << x << endl; });

This code computes the absolute value of a integer vector

First we initialize the vector with zeroes and then with ones

 //initialize vector with zeroes
vector<int> v = vector(MAX_SIZE,0);
//initialize vector with ones
transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return 1;});

Second we generate some random numbers

//get some random numbers
transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return rand()-(RAND_MAX/2);});

Third we compute the abs()

//calculate ABS
transform(execution::par,v.begin(),v.end(),v.begin(),[] (int x) -> auto { return abs(x);});

Print the vector in std out sequentially to avoid data races

for_each(execution::seq,v.begin(), v.end(), [] (int x) -> auto { cout << x << endl; });

Parallel Euclidean Distance

$\sqrt {\sum _{i=1}^{n}(x_{i}-y_{i})^{2}}$

   vector<double> v1 = { 1.0, 0.0, 0.0 };
   vector<double> v2 = { 1.0, 1.0, 0.0 };
   //euclidean distance
   auto euclidean = sqrt(transform_reduce(execution::par,v1.begin(),v1.end(),v2.begin(),
                      //initial reduce value
                      0.0,
                      //reduce function: sum
                      [] (auto x, auto y) -> auto {return x+y;},
                      //transform function: substract^2
                      [] (auto x, auto y) -> auto {return (x-y)*(x-y);}
                     ));
   cout << euclidean << endl;

First we initialize the input vectors

 vector<double> v1 = { 1.0, 0.0, 0.0 };
 vector<double> v2 = { 1.0, 1.0, 0.0 };

For every pair from x and y, compute transform (x-y)^2

[] (auto x, auto y) -> auto {return (x-y)*(x-y)

The result of the transform will be used in the parallel reduce adding all elements.
```
[] (auto x, auto y) -> auto {return x+y;}
```

Compiling our code

After having our environment installed, we need to compile our code.

CPATH=/path_to_parallelstl/include g++ -std=c++17 main.cpp -ltbb

We are forcing to use the new stl with the CPATH env var.
We are forcing C++17 standard
We are linking with Intel® TBB Threading Building Blocks

Conclusions

Learning parallel programming is too difficult because our mindset is too much attached to imperative programming.

When we start to code in functional programming, mutual exclusion and data races are less common because our data structure is independant between iterations.

We still have to be careful about thread safe algorithm. But If you use lambdas and the input and output are different in each iteration of the loop, You will end up with high performant code.

Next Steps

Nowadays, a computer is a heterogeneous system with too many processors.

Some of them are called CPU, others are called GPU and others are called FPGA.

Khronos(R) Group is investing in SYCL library. A high performance library to bring C++ 2017 code in all this devices.

Single source SYCL C++ on Xilinx FPGA
ComputeCPP: A library for execute C++ code in GPUs

I Hope you enjoy this post.

Scalable web applications with swarm

Sat, 25 Aug 2018 12:00:00 +0000

Designing horizontal scalable application can be a difficult task if you don’t know the right tools to achieve this pourpose. In this post we make use of docker swarm to create an horizontal scalable web application.

TL;DR

docker swarm init && \
git clone https://github.com/mercuriete/laravel-docker-redis.git && \
cd laravel-docker-redis && \
docker stack deploy --compose-file=docker-stack-compose.yml laravel && \
sleep 5 && \
docker service scale laravel_php=2 

Disclaimer

The scope of this post is not teaching you how to create an application in a given framework. I decided to use php and laravel because I thought was a practical approach instead of making a academic tutorial without real application in real life.

If you had never heard about docker or what is a stateless web application. This topics are out of the scope of this post.

Given that you know what are docker containers or how to create a stateless web application, you can continue, otherwise I will encourage you to read the docs.

Some docker tutorial

Some definition about Stateless App

Swarm init

First of all we need to have up and running our docker swarm cluster. In this tutorial we do the examples with a cluster of only one node. You can think this is kind of pointless but in a docker swarm adding more nodes or scaling your application is so easy that once you learn with one node, the rest is the same than with one node.

docker swarm init

output:

Swarm initialized: current node (RANDOM_STRING) is now a manager.  
To add a worker to this swarm, run the following command:  
docker swarm join --token RANDOM_STRING 192.168.1.43:2377  
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.  

With this command you are ready to deploy swarm stacks.

Cloning the repository

git clone https://github.com/mercuriete/laravel-docker-redis.git

Understanding the application

This application is a example of a php application with a mysql database and a redis database for authentication.

Nginx: Nginx is the entry point of our application. It translates all request on something php likes. See default.conf
Php: This is our application. Is made with laravel php framework. This is the component we want to scale. This application can potentially have hundreds of request that consumes lots of cpu resources.
Database: This is our persistence. We need to ensure this container always is back up in a regular basis and uses a persistent volume. How to create persistent volumes is out of scope of this tutorial.
Redis: This is the mecanism that allows the application to be Stateless. It records all session tokens in order to avoid to have any state on the php application. We don’t need to persist the state of redis, but if you lose the redis information, all your customers are logged out from your application. Be careful about how many times you restart this container.

Deploying our application

docker stack deploy --compose-file=docker-stack-compose.yml laravel

output:

Creating network laravel_default  
Creating service laravel_database  
Creating service laravel_redis  
Creating service laravel_nginx  
Creating service laravel_php  

Now we have up and running our stack. We can check it with the following command.

docker service ls

output:

ID NAME MODE REPLICAS IMAGE PORTS  
05kyuuxj36bj laravel_database replicated 0/1 mysql:5.7  
ofgn8q9rmmpw laravel_nginx replicated 0/1 mercuriete/laravel-redis-nginx:latest *:80->80/tcp  
lzdl6gdr9irr laravel_php replicated 0/1 mercuriete/laravel-redis:latest  
39anu4sg762z laravel_redis replicated 1/1 redis:4.0

Scaling our application

We need to scale our application horizontally in order to comply with the new amount of users that required more performance in our application.

docker service scale laravel_php=2

output:

laravel_php scaled to 2  
overall progress: 2 out of 2 tasks  
1/2: running [==================================================>]  
2/2: running [==================================================>]  
verify: Service converged

now our application should be like this:

Now we have 2 instances of php behind a load balancer. This balancer is called Ingress and sometimes docker call it “the routing mesh”.

You can read more documentation here

Rolling Updates

Rolling updates is a feature of docker swarm that allows you to upgrade your containers with a new release of your code with 0 downtime even in only 1 node cluster.

First we have to make a commit of our code and wait to our CI tool to recreate the image and upload It to our registry. We are using dockerhub automated builds to achieve this.

Type in the shell the followint command. Update delay is wait until the next instance is recreated. Under the hood swarm is recreating the service one by one with a delay of 30 seconds. This means we have some time the application with both versions of our application at the same time, but is up all the time.

docker service update laravel_php --update-delay 30s --image mercuriete/laravel-redis  

output:

laravel_php  
overall progress: 2 out of 2 tasks  
1/2: running [==================================================>]  
2/2: running [==================================================>]  
verify: Service converged  

Conclusions

Nowadays we have more tools to achieve horizontal scaling in web applications. I remember when I had to configure HAProxy in an 8 node distributed application. It was a nightmare.

In Addition, we can deploy our releases in a way that nobody can realise whether our application is being updated or not.

Next steps

In the following post we will be learning to use portainer to do the same tutorial but with a graphical frontend with a few clicks.

Detecting memory leaks in c++ code

Sat, 18 Aug 2018 12:00:00 +0000

Memory leaks are a very common problem in c++ development that can cause severe problems due to difficult to found the problem.

Windows version

For finding memory leaks in Microsoft Visual Studio builds we are using CRT library following the official documentation. Finding Memory Leaks Using the CRT Library.

I created a helper to make more usable that library. I will explain it line by line Memory leak detection helper

Preprocesor header

#ifndef MEMORY_LEAK_HPP
#define MEMORY_LEAK_HPP
#if defined(_WIN32) || defined(_WIN64)
#ifdef _DEBUG

We only want to generate code in windows architectures and only when debugging.

macro for new expansion

#define DBG_NEW new ( _NORMAL_BLOCK , __FILE__ , __LINE__ )
#define new DBG_NEW

This automatically expands a new sentence into a new ( _NORMAL_BLOCK , FILE , LINE )

The purpouse of this macro is to have more accurate line count in the memory leak reports

library initialization

int initialize_memory_leak_detector(){
   std::cout << "initializing memory leak detector" << std::endl;
   _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
   return 0;
}
int memory_leak_initizalization_result = initialize_memory_leak_detector();

This global variable is useless but this triggers a initialization before the main was called. Probably this sentence is called before your code is started.

This function sets debug flags according with the microsoft documentation for dump a memory leak report on exit of your application

at exit handler

void atexit_handler(){
   int result = 0;
   std::cout << "ending memory leak detector and dumping report" << std::endl;
   if ((result = _CrtDumpMemoryLeaks()) != 0) {
      std::cout << "memory leak detected. please debug with Visual Studio" << std::endl;
   }
   else{
      std::cout << "no memory leaks detected" << std::endl;
   }
   //Hahahahaha exit on an exit handler!
   //feel free to blame me if you get something really bad.
   exit(result);
}

int memory_leak_dump_report_at_exit = std::atexit(atexit_handler);

This function is registered at start of your application and is called just before you application is exiting. This dumps a memory leak report on debug logger that is not available to read in continuous integration tools. This is why we exit with an exit code to break the continous integration pipeline once we detect a memory leak.

As you can see, I am calling exit inside an at exit handler. This is totally forbidden due to a possible infinite loop but it works as I intended.

Linux & Mac version

On Unix systems ASAN library from google will be used. This library is integrated in g++ and clang compilers and can be used as easy as setting a flag during comilation.

example on github

The only thing we need to do is pass to the compiler the flags:

-g -O0 -fsanitize=leak"

-g: is for debugging symbols

-o0 is for don’t optimization for accurate line number count

-fsanitize=leak is for activate ASAN with low overhead only detecting leaks.

Then when you start you application you will need to add an enviromental variable example on travis

ASAN_OPTIONS=detect_leaks=1 ./app

with that done you will have something like this: travis build log

Direct leak of 8 byte(s) in 1 object(s) allocated from:
 #0 0x7fb8069879f8 in operator new(unsigned long) (liblsan.so.0+0xf9f8)
 #1 0x40bd8c in main /app/src/main.cpp:13
 #2 0x7fb8060a2f44 in __libc_start_main (libc.so.6+0x21f44)

then you can go to the offending line and check what is happening

string* leaking_pointer = new string("leaking object");
//You forgot to delete pointer
leaking_pointer = new string("Hello, World!");
cout << *leaking_pointer << endl;
delete leaking_pointer;

Is really obvious whats going on. You forgot to delete a pointer before reuse for another instance.

Conclusions

Given that I have a bias about linux vs windows, Is obvious to understand that on the windows counterpart you have to modify your code a little bit in order to enable memory sanitizer tools. With ASAN you only need to compile you code with debug symbols and ASAN will take care of analyze the heap for you.

Summarizing, there are some tools that can improve our code quality with a little bit of effort. This days are not acceptable to have such a problem comparing with days when instrumenting your application were more difficult.