libssh is running in the Matrix now

Flattr this!

Since I joined the libssh project we started to write tests to find regression and make development easier. This has been achieved using the a unit testing framework called cmocka which I maintain and develop. The problem is that to run these tests you need to modify the sshd configuration and setup a test user so that the tests can be successfully executed. This is something contributors normally don’t do so we need to rely on our testing infrastructure.

In 2013 I’ve started the cwrap project. cwrap is a set of tool to make full network server/client testing made easy. These tools are used to make it possible to run the Samba Testsuite easily on every machine without setting anything up. Some time ago I’ve started to use cwrap for libssh testing. Finally I found the time to finish the task.

libssh in the Matrix

Now a libssh client tests sets up an artificial test environment. We have a passwd, shadow and group file so we can use two users to authenticate (nss_wrapper). sshd is running as the user starting the testcase but as it is part of the Matrix it thinks it is root (uid_wrapper). The client and server think they communicate on a real network (socket_wrapper) but it is again the Matrix!

It took me a while to get it working and I needed to implement new feature to the wrapper libraries of cwrap. socket_wrapper needed support to report TCP_NODELAY in getsockopt(). nss_wrapper needed shadow file support for password authentication so I had to add support for getspnam(). And as sshd is paranoid uid_wrapper needed checks if if is privileged to actually change to the user. After it drops privileged it checks if it really can’t go back.

With all of this implemented and new releases of the wrappers, which I’m preparing at the moment, all you have to do is to install cmocka, socket_wrapper, nss_wrapper and uid_wrapper and run ‘make test’. The Matrix will be created and libssh tested. You can find the cwrap libssh branch here.

There is one test for a feature missing right now. We do not test keyboard-interactive authentication, but the cwrap project is working on a new wrapper to fix this. Stay tuned!

Hunting down a fd closing bug in Samba

Flattr this!

In Samba I had a failing test suite. I have nss_wrapper compiled with debug messages turned on, so it showed me the following line:

NWRAP_ERROR(23052) - nwrap_he_parse_line: 3 Invalid line[TDB]: 'DB'

The file should parse a hosts file like /etc/hosts, but the debug line showed that it tried to parse a TDB (Trivial Database) file, Samba database backend. I’ve started to investigate it and wondered what was going on. This morning I called Michael Adam and we looked into the issue together. It was obvious that something closed the file descriptor for the hosts file of nss_wrapper and it was by Samba to open other files. The big question was, what the heck closes the fd. As socket_wrapper was loaded and it wraps the open() and close() call we started to add debug to the socket_wrapper code.

So first we added debug statements to the open() and close() calls to see when the fd was opened and closed. After that we wanted to see a stacktrace at the close() call to see what is the code path were it happens. Here is the code how to do this:

commit 6c632a4419b6712f975db390145419b008442865
Author:     Andreas Schneider 
AuthorDate: Thu Mar 26 11:07:38 2015 +0100
Commit:     Andreas Schneider 
CommitDate: Thu Mar 26 11:07:59 2015 +0100

    DEBUG stacktrace
 src/socket_wrapper.c | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/src/socket_wrapper.c b/src/socket_wrapper.c
index 1188c4e..cb73cf2 100644
--- a/src/socket_wrapper.c
+++ b/src/socket_wrapper.c
@@ -80,6 +80,8 @@
 #include <rpc/rpc.h>
+#include <execinfo.h>
 enum swrap_dbglvl_e {
@@ -303,8 +305,8 @@ static void swrap_log(enum swrap_dbglvl_e dbglvl,
 		switch (dbglvl) {
-					"SWRAP_ERROR(%d) - %s: %s\n",
-					(int)getpid(), func, buffer);
+					"SWRAP_ERROR(ppid=%d,pid=%d) - %s: %s\n",
+					(int)getppid(), (int)getpid(), func, buffer);
 			case SWRAP_LOG_WARN:
@@ -565,10 +567,35 @@ static int libc_bind(int sockfd,
 	return swrap.fns.libc_bind(sockfd, addr, addrlen);
 static int libc_close(int fd)
 	swrap_load_lib_function(SWRAP_LIBC, close);
+	if (fd == 21) {
+		void *backtrace_stack[BACKTRACE_STACK_SIZE];
+		size_t backtrace_size;
+		char **backtrace_strings;
+		SWRAP_LOG(SWRAP_LOG_ERROR, "fd=%d", fd);
+		backtrace_size = backtrace(backtrace_stack,BACKTRACE_STACK_SIZE);
+		backtrace_strings = backtrace_symbols(backtrace_stack, backtrace_size);
+			  "BACKTRACE %lu stackframes",
+			  (unsigned long)backtrace_size);
+		if (backtrace_strings) {
+			size_t i;
+			for (i = 0; i < backtrace_size; i++) {
+					" #%lu %s", i, backtrace_strings[i]);
+			}
+		}
+	}
 	return swrap.fns.libc_close(fd);
@@ -704,6 +731,8 @@ static int libc_vopen(const char *pathname, int flags, va_list ap)
 	fd = swrap.fns.libc_open(pathname, flags, (mode_t)mode);
+	SWRAP_LOG(SWRAP_LOG_ERROR, "path=%s, fd=%d", pathname, fd);
 	return fd;

We found out that the code responsible for this created a pipe() to communitcate with the child and then forked. The child called close() on the second pipe file descriptor. So when another fork happend in the child, the close() on the pipe file descriptor was called again and we closed a fd of the process to a tdb, connection or something like that. So initializing the pipe fd array with -1 and only calling close() if we have a file description which is not -1, fixed the problem.

If you need a better stacktrace you should use libunwind. However socket_wrapper can be a nice little helper to find bugs with file descriptors 😉

BUG: Samba standard process model closes random files when forking more than once

cmocka 1.0

Flattr this!

At the beginning of February I attended in Brno and the days before I had a hack week with Jakub Hrozek on cmocka. cmocka is a unit testing framework for C with support for mock objects.

We already rewrote the test runner last year and it was time to finish it and add support for several different message output formats. You are able to switch between cmocka standard output, Subunit, Test Anything Protocol and jUnit XML reports. In addition we we have a skip() function and test_realloc() to detect buffer overflows and memory leaks now.

You can find all other required information on the overhauled shiny new website:

New uid_wrapper with full threading support.

Flattr this!

Today I’ve released a new version of uid_wrapper (1.1.0) with full threading support. Robin Hack a colleague of mine spent a lot of time improving the code and writing tests for it. It now survives funny things like forking in a thread. We also added two missing functions and fixed several bugs. uid_wrapper is a tool to help you writing tests for your application.

If you don’t know uid_wrapper and wonder what you can do with it, here is an example:

$ id
uid=1000(asn) gid=100(users) groups=100(users),478(docker)
uid=0(root) gid=0(root) groups=0(root)

More details about uid_wrapper can be found on the cwrap project website, here.

Taking your bike on a plane

Flattr this!

Here is a totally computer unrelated post! Several people asked me how do I protect my bike to transport it safely on a plane. One possibility is to use a bike box, but the issue with a box is that airline personal likes big boxes, because they can use it to pile a lot of suitcases on it. I prefer to just wrap it with cardboard! Normally I go to a supermarket and ask if they have some cardboard for me. I’m sure they are happy to get rid of some. What you need bring from home in addition is a piece of rope, zip ties, duct tape, an old towel and a multitool or cutter.

I prepare everything at the supermarket. Cut the cardboard for the different pieces of the bike, put holes in the cardboard for the zip ties (first put duct tape on the cardboard then make the hole trough the duct tape and the cardboard!). Make sure you can still push the bike, the wheels should turn around. In the end I have a small package like this:

Bike on a plane, cardboard collection

It is easy to transport. Either on the back of the bike or on your back 😉

At the airport you remove the pedals and fix the crank. Put the towel over the saddle and fix it with duct tape or a piece of rope. Tie a rope from the saddle to the handle bar so you can carry it. This makes it also easier for the airport personal to lift it. Then cover the bike with cardboard. Some parts are fixed on the bike with zip ties so they can’t move. In the end it normally looks like this:

Bike on a plane, protected with cardboard

If you’re on a bike trip you normally have 4 bike panniers with you, but the airline only allows to carry one luggage. Either the airport offers to wrap them or you go to a grocery store and buy plastfoil for food. It is very thin so you need 60m. It is not really Eco-friendly but the only way I know. Suggestions are welcome.

First start to wrap the two biggest panniers:

bike on a plane, panniers.

I use a rope to connect them and make sure not to loose one. After the two are in a good shape (ca. 25m) put the smaller panniers on the side and start to wrap the whole thing:

bike on a plane, wrapped panniers

Have a safe trip!

resolv_wrapper 1.0.0 – the new cwrap tool

Flattr this!

I’ve released a new preloadable wrapper named resolv_wrapper which can be used for nameserver redirection or DNS response faking. It can be used in testing environment to route DNS queries to a real nameserver separate from resolv.conf or fake one with simple config file. We tested it on Linux, FreeBSD and Solaris. It should work on other UNIX flavors too.


You can download resolv_wrapper here.

A talk about cwrap at LinuxCon Europe

Flattr this!

Next week is the LinuxCon Europe in Düsseldorf, Germany. I will be there and give a talk about cwrap, a set of tools to make client/server testing easy on a single machine. Testing network applications correctly is hard. This talk will demonstrate how to create a fully isolated network environment for client and server testing on a single host, complete with synthetic account information, hostname resolution, and privilege separation.

I hope you will attend my talk if you are there. If you can’t attend the LinuxCon Europe, but you’re going to the Linux Plumbers Conference then say hello and lets talk about cwrap there!

At the LinuxCon Europe I will announce new cool stuff and the website will be updated. So you should check

next week!

cwrap talk

How to get real DNS resolving in ‘make test’?

Flattr this!

As you might know I’m working (hacking) on Samba. Samba has a DNS implementation to easier integrate all the AD features. The problem is we would like to talk to the DNS server but /etc/resolv.conf points to a nameserver so your machine is correctly working in your network environment. For this Samba in our dns resolver library we implemented a way to setup a dns_hosts_file to fake DNS queries. This works well for binaries provided by Samba but not for 3rd party application. As Günther Deschner and I are currently working on MIT Kerberos support the libkrb5 library always complained that it is not able to talk query the DNS server to find the KDC. So it was time to really fix this!

I’ve sat down and did some research how we get this working. After digging through the glibc code, first I thought we could redirect the fopen(“/etc/resolv.conf”) call. Well as this is called in a glibc internal function it directly calls _IO_fopen() which isn’t weak symbol. So I looked deeper and recognized that I have access to the resovler structure which holds the information to the nameserver. I could simply modify this!

It was time to implement another wrapper, resolv_wrapper. Currently it only wraps the functions required by Samba and MIT Kerberos, res_(n)init(), res_(n)close, res_(n)query and res_(n)search. With this I was able to run kinit which asks the DNS server for a SRV record to find the KDC and it worked. With Jakub Hrozek I cleaned up the code yesterday and we created a parser for a resolv.conf file.

Here is a tcpdump of the kinit tool talking to the DNS server with socket_wrapper over IPv6.

resolv_wrapper will be available on soon!