Next week is the LinuxCon Europe in Düsseldorf, Germany. I will be there and give a talk about cwrap, a set of tools to make client/server testing easy on a single machine. Testing network applications correctly is hard. This talk will demonstrate how to create a fully isolated network environment for client and server testing on a single host, complete with synthetic account information, hostname resolution, and privilege separation.
I hope you will attend my talk if you are there. If you can’t attend the LinuxCon Europe, but you’re going to the Linux Plumbers Conference then say hello and lets talk about cwrap there!
At the LinuxCon Europe I will announce new cool stuff and the website will be updated. So you should check
As you might know I’m working (hacking) on Samba. Samba has a DNS implementation to easier integrate all the AD features. The problem is we would like to talk to the DNS server but /etc/resolv.conf points to a nameserver so your machine is correctly working in your network environment. For this Samba in our dns resolver library we implemented a way to setup a dns_hosts_file to fake DNS queries. This works well for binaries provided by Samba but not for 3rd party application. As Günther Deschner and I are currently working on MIT Kerberos support the libkrb5 library always complained that it is not able to talk query the DNS server to find the KDC. So it was time to really fix this!
I’ve sat down and did some research how we get this working. After digging through the glibc code, first I thought we could redirect the fopen(“/etc/resolv.conf”) call. Well as this is called in a glibc internal function it directly calls _IO_fopen() which isn’t weak symbol. So I looked deeper and recognized that I have access to the resovler structure which holds the information to the nameserver. I could simply modify this!
It was time to implement another wrapper, resolv_wrapper. Currently it only wraps the functions required by Samba and MIT Kerberos, res_(n)init(), res_(n)close, res_(n)query and res_(n)search. With this I was able to run kinit which asks the DNS server for a SRV record to find the KDC and it worked. With Jakub Hrozek I cleaned up the code yesterday and we created a parser for a resolv.conf file.
Here is a tcpdump of the kinit tool talking to the DNS server with socket_wrapper over IPv6.
resolv_wrapper will be available on cwrap.org soon!
by Jakub Hrozek and Andreas Schneider
The LD_PRELOAD trick!
Preloading is a feature of the dynamic linker (ld). It is a available on most Unix system and allows to load a user specified, shared library before all other shared libraries which are linked to an executable.
Library pre-loading is most commonly used when you need a custom version of a library function to be called. You might want to implement your own
free(3) functions that would perform a rudimentary leak checking or memory access control for example, or you might want to extend the I/O calls to dump data when reverse engineering a binary blob. In this case, the library to be preloaded would implement the functions you want to override with prelinking. Only functions of dynamically loaded libraries can be overridden. You’re not able to override a function the application implements by itself or links statically with.
The library to preload is defined by the environment variable
LD_PRELOAD, such as
LD_PRELOAD=libwurst.so. The symbols of the preloaded library are bound first, before other linked shared libraries.
Lets look into symbol binding in more details. If your application calls a function, then the linker looks if it is available in the application itself first. If the symbol is not found, the linker checks all preloaded libraries and only then all the libraries which have been linked to your application. The shared libraries are searched in the order which has been given during compilation and linking. You can find out the linking order by calling
'ldd /path/to/my/applicaton'. If you’re interested how the linker is searching for the symbols it needs or if you want do debug if the symbol of your preloaded library is used correctly, you can do that by enabling tracing in the linker.
A simple example would be
'LD_DEBUG=symbols ls'. You can find more details about debugging with the linker in the manpage:
Your application uses the function
- Your application doesn’t implement it.
- The linked libc.so provides
open(2) symbol from
libcwrap.so gets bound!
The wrappers used for creating complex testing environments of the cwrap project use preloading to supply their own variants of several system or library calls suitable for unit testing of networked software or privilege separation. For example, one wrapper includes its version of most of the standard API used to communicate over sockets that routes the communication over local sockets.
Several people asked me about the status about the Active Directory Domain Controller support of Samba in Fedora. As Fedora and RHEL are using MIT Kerberos implementation as its Kerberos infrastructure of choice, the Samba Active Directory Domain Controller implementation is not available with MIT Kereberos at the moment. But we are working on it!
Günther Deschner and I gave a talk at the SambaXP conference about our development efforts in this direction:
The road to MIT KRB5 support
I hope this helps to understand that this is a huge task.
Maybe you already heard of the cwrap project. A set of tools to create a fully isolated network environment to test client/server components on a single host. socket_wrapper is a part of cwrap and I released version 1.1.0 today. In this release I worked together with Michael Adam and we implemented some nice new features like support for IP_PKTINFO for binding on UDP sockets, bindresvport() and more socket options via getsockopt(). This was mostly needed to be able to create a test environment for MIT Kerberos.
The upcoming features for the next version are support for passing file description between processes using a unix domain socket and sendmsg()/recvmsg() (SCM_RIGHTS). We would also like to make socket_wrapper thread-safe.
Together with Jakub Hrozek I wrote an article about cwrap which is a set of tools to test your full software stack on a single machine. The article is open to the public now.
Read the article …
Last Friday I’ve released cmocka 0.4.0. It has several bugfixes and at least two new features. One is support for groups. This means you can define a setup and teardown function for a group of unit tests. I think some people have been waiting for this.
You can find an example here. It is simple and easy to use.
The other small feature is a new macro:
assert_return_code(). It is designed for standard C function return values which return 0 for success and less than 0 to indicate an error with errno set. It will produce a nice error message! The rest are bugfixes and improvements for error message.
Thanks to all contributor and bug reporter!
If you think cmocka is a great piece of software, please vote it up on stackoverflow, thanks.
After the Update to Fedora 20 I forgot to update the linker to Gold. Today I released that linking Samba is horribly slow. Time to change the linker to Gold again:
/usr/sbin/alternatives --set ld /usr/bin/ld.gold
/usr/bin/update-alternatives --set ld /usr/bin/ld.gold
on one single machine!
FOSDEM/Brussels, February 2nd, I gave a talk about cwrap. I announced and released version 1.0.0 of cwrap, a set of tools to create a fully isolated network environment to test client/server components on a single host.
It provides synthetic account information, hostname resolution and privilege separation support. The heart of cwrap consists of three libraries you can preload to any executable.
The libc wrapper project does not require virtualization and can be used to build environments on different operating systems. The project
consists of a socket wrapper, NSS module wrapper (users, groups, hosts), and a (s)uid wrapper with support for GNU/Linux, BSD and Solaris.
The origin of these wrappers is the Samba project, where the wrappers are in use since many years to successfully test their protocol implementations. Now it is possible to use them outside of the Samba project. The wrappers have been improved and are loaded with new features.
Learn more at http://cwrap.org/
Next weekend, February 1st and 2nd, will be the FOSDEM conference in Brussels again. I will be there and give a talk in the Testing and Automation devroom. I hope you will be there and watch my presentation on Sunday, February 2nd, 11:30, Room: UD2.218A.
See you there!