Samba 4.7.0 (Samba AD for the Enterprise)

Enterprise distributions like Red Hat or SUSE are required to ship with MIT Kerberos. The reason is that several institutions or governments have a hard requirement for a special Kerberos implementation. It is the reason why the distributions by these vendors (Fedora, RHEL, openSUSE, SLES) only package Samba FS and not the AD component.

To get Samba AD into RHEL some day it was clear, that we need to port it to MIT Kerberos.

In 2013 we started to think about this. The question which arise first was: How do we run the tests if we port to MIT Kerberos? We want to start the krb5kdc daemon. This was more or less the birth of the cwrap project! Think of cwrap like it is “The Matrix” where reality is simulated and everything is a lie. It allows us to create an artificial environment emulating a complete network to test Samba. It took nearly a year till we were able to integrate the first part of cwrap, socket_wrapper, into Samba.

Then the work to port Samba AD to MIT Kerberos started. We created a simple abstraction of Samba KDC routines so we could convert between Heimdal and MIT Kerberos. Then created a MIT KDB module and were able to start the krb5kdc process.

In 2015 we had more than 140 patches for Samba AD ready and pushed most of them upstream in April. We still had 70 testsuites failing. We started to implement missing features and fixed tests to work with MIT Kerberos tools. During that time we often had setbacks because features in MIT Kerboros were missing which we required. So we started to implement missing features in MIT Kerberos.

In September of 2015 we started to implement missing pieces in ‘samba-tool’ to provision a domain with MIT Kerberos involved. Till the end of the year we implemented the backup key protocol using GnuTLS (which also needed to add features for us first).

From January till July 2016 we implemented more features in MIT Kerberos to get everything working. In August we had most of the stuff working just the trust support wasn’t working. From there we discovered bug after bug in the implementation how trusts are handled and fixed bug by bug. We had to do major rewrites of code in order to get everything working correctly. The outcome was great. We improved our trust code and got MIT Kerberos working in the end.


That’s the day when I pushed the final patchset to our source code repository!

It took Günther Deschner, Stefan Metzmachen and me more than 4 years to implement Samba AD with MIT Kerberos. Finally with the release Samba 4.7.0 it is available in a to use for everyone.

Fedora 27 will be the first version with Samba AD.

Microsoft Catalog Files and Digital Signatures decoded

TL;DR: Parse and print .cat files: parsemscat


Günther Deschner and myself are looking into the new Microsoft Printing Protocol [MS-PAR]. Printing always means you have to deal with drivers. Microsoft package-aware v3 print drivers and v4 print drivers contain Microsoft Catalog files.

A Catalog file (.cat) is a digitally-signed file. To be more precise it is a PKCS7 certificate with embedded data. Before I started to look into the problem understanding them I’ve searched the web, if someone already decoded them. I found a post by Richard Hughes: Building a better catalog file. Richard described some of the things we already discovered and some new details. It looks like he gave up when it came down to understand the embedded data and write an ASN.1 description for it. I started to decode the myth of Catalog files the last two weeks and created a tool for parsing them and printing what they contain, in human readable form.


The embedded data in the PKCS7 signature of a Microsoft Catalog is a Certificate Trust List (CTL). Nikos Mavrogiannopoulos taught me ASN.1 and helped to create an ASN.1 description for the CTL. With this description I was able to start parsing Catalog files.



CatalogNameValue ::= SEQUENCE {
    name       BMPString, -- UCS2-BE
    flags      INTEGER,
    value      OCTET STRING -- UCS2-LE




The PKCS7 part of the .cat-file is the signature for the CTL. Nikos implemented support to get the embedded raw data from the PKCS7 Signature with GnuTLS. It is also possible to verify the signature using GnuTLS now!
The CTL includes members and attributes. A member holds information about file name included in the driver package, OS attributes and often a hash for the content of the file name, either SHA1 or SHA256. I’ve written abstracted function so it is possible to create a library and a simple command line tool called dumpmscat.

Here is an example of the output:

  CHECKSUM: E5221540DC4B974F54DB4E390BFF4132399C8037

  FILE: sambap1000.inf, FLAGS=0x10010001
  OSATTR: 2:6.0,2:6.1,2:6.4, FLAGS=0x10010001
  MAC: SHA1, DIGEST: E5221540DC4B974F54DB4E39BFF4132399C8037

In addition the CTL has normally a list of attributes. In those attributes are normally OS Flags, Version information and Hardware IDs.

  NAME=OS, FLAGS=0x10010001, VALUE=VistaX86,7X86,10X86
  NAME=HWID1, FLAGS=0x10010001, VALUE=usb\\vid_0ff0&pid_ff00&mi_01

Currently the projects only has a command line tool called: dumpmscat. And it can only print the CTL for now. I plan to add options to verify the signature, dump only parts etc. When this is done I will create a library so it can easily be consumed by other software. If someone is interested and wants to contribute. Something like signtool.exe would be nice to have.

A new cmocka release version 1.1.0

It took more than a year but finally Jakub and I released a new version of cmocka today. If you don’t know it yet, cmocka is a unit testing framework for C with support for mock objects!

We set the version number to 1.1.0 because we have some new features:

  • Support to catch multiple exceptions
  • Support to verify call ordering (for mocking)
  • Support to pass initial data to test cases
  • A will_return_maybe() function for ignoring mock returns
  • Subtests for groups using TAP output
  • Support to write multiple XML output files if you have several groups in a test
  • and improved documentation

We have some more features we are working on. I hope it will not take such a long time to release them.

Testing PAM modules and PAM-aware applications in the Matrix

Jakub Hrozek and I are proud to announce the first release of pam_wrapper. This tool allows you to either simplify testing PAM modules or your application using PAM to authenticate users. PAM (Pluggable Authentication Modules) is a layer of abstraction on top of Unix authentication.

For testing PAM-aware applications we have written a simple PAM module called pam_matrix. If you plan to test a PAM module you can use the pamtest library we have implemented. It simplifies testing of modules. You can combine it with the cmocka unit testing framework or you can use the provided Python bindings to write tests for your module in Python.

Jakub and I have written an article for to provide more details how to use it. You can find it here.

Now start testing your PAM module or application!

libssh is running in the Matrix now

Since I joined the libssh project we started to write tests to find regression and make development easier. This has been achieved using the a unit testing framework called cmocka which I maintain and develop. The problem is that to run these tests you need to modify the sshd configuration and setup a test user so that the tests can be successfully executed. This is something contributors normally don’t do so we need to rely on our testing infrastructure.

In 2013 I’ve started the cwrap project. cwrap is a set of tool to make full network server/client testing made easy. These tools are used to make it possible to run the Samba Testsuite easily on every machine without setting anything up. Some time ago I’ve started to use cwrap for libssh testing. Finally I found the time to finish the task.

libssh in the Matrix

Now a libssh client tests sets up an artificial test environment. We have a passwd, shadow and group file so we can use two users to authenticate (nss_wrapper). sshd is running as the user starting the testcase but as it is part of the Matrix it thinks it is root (uid_wrapper). The client and server think they communicate on a real network (socket_wrapper) but it is again the Matrix!

It took me a while to get it working and I needed to implement new feature to the wrapper libraries of cwrap. socket_wrapper needed support to report TCP_NODELAY in getsockopt(). nss_wrapper needed shadow file support for password authentication so I had to add support for getspnam(). And as sshd is paranoid uid_wrapper needed checks if if is privileged to actually change to the user. After it drops privileged it checks if it really can’t go back.

With all of this implemented and new releases of the wrappers, which I’m preparing at the moment, all you have to do is to install cmocka, socket_wrapper, nss_wrapper and uid_wrapper and run ‘make test’. The Matrix will be created and libssh tested. You can find the cwrap libssh branch here.

There is one test for a feature missing right now. We do not test keyboard-interactive authentication, but the cwrap project is working on a new wrapper to fix this. Stay tuned!

Hunting down a fd closing bug in Samba

In Samba I had a failing test suite. I have nss_wrapper compiled with debug messages turned on, so it showed me the following line:

NWRAP_ERROR(23052) - nwrap_he_parse_line: 3 Invalid line[TDB]: 'DB'

The file should parse a hosts file like /etc/hosts, but the debug line showed that it tried to parse a TDB (Trivial Database) file, Samba database backend. I’ve started to investigate it and wondered what was going on. This morning I called Michael Adam and we looked into the issue together. It was obvious that something closed the file descriptor for the hosts file of nss_wrapper and the samba binary opend a different file getting the same fd number assigned. The big question was, what the heck closes the fd. As socket_wrapper was loaded and it wraps the open() and close() call we started to add debug to the socket_wrapper code.

So first we added debug statements to the open() and close() calls to see when the fd was opened and closed. After that we wanted to see a stacktrace at the close() call to see what is the code path were it happens. Here is the code how to do this:

commit 6c632a4419b6712f975db390145419b008442865
Author:     Andreas Schneider < gro.a1506262386bmas@1506262386nsa1506262386>
AuthorDate: Thu Mar 26 11:07:38 2015 +0100
Commit:     Andreas Schneider < gro.a1506262386bmas@1506262386nsa1506262386>
CommitDate: Thu Mar 26 11:07:59 2015 +0100

    DEBUG stacktrace
 src/socket_wrapper.c | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/src/socket_wrapper.c b/src/socket_wrapper.c
index 1188c4e..cb73cf2 100644
--- a/src/socket_wrapper.c
+++ b/src/socket_wrapper.c
@@ -80,6 +80,8 @@
 #include <rpc/rpc.h>
+#include <execinfo.h>
 enum swrap_dbglvl_e {
@@ -303,8 +305,8 @@ static void swrap_log(enum swrap_dbglvl_e dbglvl,
 		switch (dbglvl) {
-					"SWRAP_ERROR(%d) - %s: %s\n",
-					(int)getpid(), func, buffer);
+					"SWRAP_ERROR(ppid=%d,pid=%d) - %s: %s\n",
+					(int)getppid(), (int)getpid(), func, buffer);
 			case SWRAP_LOG_WARN:
@@ -565,10 +567,35 @@ static int libc_bind(int sockfd,
 	return swrap.fns.libc_bind(sockfd, addr, addrlen);
 static int libc_close(int fd)
 	swrap_load_lib_function(SWRAP_LIBC, close);
+	if (fd == 21) {
+		void *backtrace_stack[BACKTRACE_STACK_SIZE];
+		size_t backtrace_size;
+		char **backtrace_strings;
+		SWRAP_LOG(SWRAP_LOG_ERROR, "fd=%d", fd);
+		backtrace_size = backtrace(backtrace_stack,BACKTRACE_STACK_SIZE);
+		backtrace_strings = backtrace_symbols(backtrace_stack, backtrace_size);
+			  "BACKTRACE %lu stackframes",
+			  (unsigned long)backtrace_size);
+		if (backtrace_strings) {
+			size_t i;
+			for (i = 0; i < backtrace_size; i++) {
+					" #%lu %s", i, backtrace_strings[i]);
+			}
+		}
+	}
 	return swrap.fns.libc_close(fd);
@@ -704,6 +731,8 @@ static int libc_vopen(const char *pathname, int flags, va_list ap)
 	fd = swrap.fns.libc_open(pathname, flags, (mode_t)mode);
+	SWRAP_LOG(SWRAP_LOG_ERROR, "path=%s, fd=%d", pathname, fd);
 	return fd;

We found out that the code responsible for this created a pipe() to communitcate with the child and then forked. The child called close() on the second pipe file descriptor. So when another fork happend in the child, the close() on the pipe file descriptor was called again and we closed a fd of the process to a tdb, connection or something like that. So initializing the pipe fd array with -1 and only calling close() if we have a file description which is not -1, fixed the problem.

If you need a better stacktrace you should use libunwind. However socket_wrapper can be a nice little helper to find bugs with file descriptors 😉

BUG: Samba standard process model closes random files when forking more than once

cmocka 1.0

At the beginning of February I attended in Brno and the days before I had a hack week with Jakub Hrozek on cmocka. cmocka is a unit testing framework for C with support for mock objects.

We already rewrote the test runner last year and it was time to finish it and add support for several different message output formats. You are able to switch between cmocka standard output, Subunit, Test Anything Protocol and jUnit XML reports. In addition we we have a skip() function and test_realloc() to detect buffer overflows and memory leaks now.

You can find all other required information on the overhauled shiny new website:

New uid_wrapper with full threading support.

Today I’ve released a new version of uid_wrapper (1.1.0) with full threading support. Robin Hack a colleague of mine spent a lot of time improving the code and writing tests for it. It now survives funny things like forking in a thread. We also added two missing functions and fixed several bugs. uid_wrapper is a tool to help you writing tests for your application.

If you don’t know uid_wrapper and wonder what you can do with it, here is an example:

$ id
uid=1000(asn) gid=100(users) groups=100(users),478(docker)
uid=0(root) gid=0(root) groups=0(root)

More details about uid_wrapper can be found on the cwrap project website, here.