Welcome to the SAMBA Domain: 10 years on

The 10th anniversary of "Welcome to the SAMBA Domain" quietly passed on Thursday 28th of August, 2007, without acknowledgement. When Paul Ashton and Luke Leighton initially reverse-engineered and published the NT Domains protocol, the floodgates were opened to both Free Software and Proprietary CIFS vendors to interoperate with Microsoft's flagship product, Windows NT, at an unprecedented level.

The anticipated reprisals from Microsoft did not happen: incredibly, instead, quiet mutual respect and cooperation crystallised the CIFS protocols into formal specifications (some of which were quietly handed out, whilst others had to wait until they were prised from Microsoft's fingers by the U.S. Dept of Justice and the E.U Commission).

In the intervening eleven years, an enormous amount has been achieved, yet, frustratingly, an enormous amount has not. This article outlines the accomplishments to date, and highlights the incredible things that could be achieved if some specific strategic inter-project free software cooperation took place. Also outlined are some hints as to how that can be accomplished, citing examples of prior proven work in which it has already been achieved, but not yet adopted.

SMB was introduced in fits and starts by numerous companies and organisations. It has a potted fragmented history that includes X-Open Specifications dating back to 1992; IBM providing an emulation layer of the underlying NetBIOS transport on top of TCP/IP, for both the UDP-equivalent (NetBIOS Datagrams - port 138) and the TCP-equivalent (NetBIOS streams - port 139). IBM had a further crack at SMB, with LANMAN 2.0; cross-licensing agreements between IBM and Microsoft resulted in that ending up in Windows and OS/2, and then Microsoft dropped IBM cold, with NT LANMAN 0.12 for Windows NT, with Windows NT set to become Microsoft's flagship operating system.

World-wide, Man-millenia of work has gone into reimplementing SMB, the transport layer and the services that run on top of it, by companies such as Microsoft, IBM, Network Appliances, UNISYS to name but a few, and groups such as the Samba and Samba TNG teams.

The reasons are simple: interoperability. There is no choice. If you want to interoperate in a Windows-dominated world; if you want to compile even the simplest of Win32 applications that will run seamlessly across many platforms, and in particular if you want the application to authenticate, the underlying infrastructure is forced to provide the authentication, transparently, for you, and that automatically draws in man-centuries of development effort, without your knowledge.

MSRPC was introduced in Windows NT 3.1 as part of the core infrastructure. The entire Operating System's critical services were written as MSRPC services. Key to this decision was the fact that Paul Leach, co-founder of Apollo, was the architect of NCA - Network Computing Architecture, and Apollo was one of the contributors to DCE/RPC. Freed from the constraints of The Open Group committee process, Microsoft was able to further enhance DCE/RPC to add critical necessary strategic enhancements, such as UCS-2 Unicode String support to NDR, and from there turn their eyes towards accidental world domination.

DCOM was announced in 1996 for Windows NT 4 and Windows 95, on top of prior releases of COM. Given that COM cannot really exist without the D, it's likely that the much despised Windows 95 team got their grubby little mitts on DCOM and, lacking any proper MSRPC infrastructure, simply ripped the D off the front. With the subsequent release of DCOM95.EXE, and the inclusion of DCOM in Windows NT 4.0 by default, the announcement of DCOM turned the computing world on its head, stifling competition and sucking developers further into the Win32 proprietary trap. Microsoft itself went on to make DCOM the basis of its major applications up until .NET attempted to take over. A quick scan through Wine's source code looking for "import stdole32.tlb" is a particularly enlightening exercise.

Interestingly, .NET itself is not linked, in any way, to DCOM (or MSRPC), which was clearly a strategy decision on Microsoft's part. Unfortunately, however, access to .NET services requires NTLM / NT authentication, and that is entirely implemented in MSRPC services.

.NET started out as a framework on top of a "Common Language Runtime". After the failure of efforts to push DCOM into the Unix and Linux worlds (with Software AG withdrawing their free DCOM product in 2004, and COMsource being available for a mere $USD 30,000), it can be surmised that Microsoft endeavoured to take a new tack. Visual BASIC, the only dynamically-typed scripting language that Microsoft had ever known, wasn't exactly taking the world by storm despite its seamless DCOM integration, and Python for Win32 which now includes DCOM support, was a long long way off.

So, Microsoft started again, with .NET, with a vision to provide support for many languages (even COBOL), and many platforms. The networked services use SOAP as the RPC mechanism, which can be dynamically-created and does not require a compiler such as MIDL.EXE - a distinct advantage. The specification for SOAP cross-references the DCE/RPC specification as inspiration for data types. SOAP is an abominally chatty and XML-based RPC mechanism that utilises HTTP as its transport. Authenticated HTTP on Microsoft's IIS Server utilises NTLMSSP. NTLMSSP utilises Microsoft's MSRPC-based NT Domain Authentication Services and Kerberos Authentication Services, and in this very indirect way it can be seen that .NET still depends on MSRPC, as do all key strategic Microsoft Technologies.

So, whilst it appears on the face of it, from looking at the MSDN library that .NET does not require MSRPC, thanks to authentication, full interoperability most definitely does require MSRPC and associated SMB and NetBIOS infrastructure (an additional man-century of coding effort: see "Samba Project Cost").

Some fifteen to twenty years later, Free Software Operating Systems are the only Operating Systems that have neither DCE/RPC nor DCOM at the heart of their infrastructure, or even any complete and self-consistent developer infrastructure or tool chains, in any form. Due to the lack of decent RPC and ORBs, attempts to recreate RPC and Inter-Process mechanisms and Object Request Brokers have resulted in D-Bus (a bastardised son of a whore for an IPC mechanism); KDE's home-grown inter-process communication mechanism DCOP (which is famously and proudly quoted as having been written in 20 minutes flat); Glib / Gobject (a highly successful but limited Object-Orientated interface, with startling similarities to COM, that cannot cross inter-process or inter-machine boundaries) and Gnome's "ORB", which uses CORBA, thanks to AT & T / Olivetti Lab's 1996 release of omniORB under the LGPL.

The strategic importance and power of DCOM and DCE/RPC is drastically and dramatically underestimated by the entire Free Software community. This article will outline the progress from the early beginnings - attempts made by Luke Leighton to address the balance - through to the current status of work achieved to date, starting right at the heart of the matter: Windows NT Authentication.

It will be seen that the goal was always to provide full network interoperability, but later on it became clear that Wine's infrastructure might even allow for developers to be seamlessly weaned off of dependence on the proprietary MSVC compilers, as well.

August 1997: "NT domain member to domain controller authentication protocol"

The "Green Light" for starting the NT Domains Research was Microsoft's release of NT 4.0 Beta. Within eight weeks, the critical first research was published in several forums, in a flurry of erroneous postings, by an over-keen and nervous Luke Leighton and an amused and laid-back Paul Ashton. Linus Nordberg followed up with a clean-room implementation in c of the published specification, which became the basis of Samba's NT Domain Primary Domain Controller functionality, and was re-published as crypt.txt

The significance of this breakthrough cannot be underestimated. Up until then, the only way to connect to a Windows NT Server was to use plain-text passwords! Windows NT 4.0 planned to disable plaintext passwords by default, which was a good idea, but Microsoft had absolutely no plans to reveal how to interoperate with the NT Domains infrastructure.

The design of NT Domains is conceptually similar to that of NIS+, Kerberos and DCE/DFS "Cells", all rolled into one, and it's no coincidence that later, in NT 5.0 (aka Windows 2000), both Kerberos and Directory Services were brought into the mix (in Microsoft's own inimicable way). Machines are securely joined to the Domain, and users can only log in from those machines (or from machines in a Domain where "Inter-Domain Trust" has been established).

The whole NT Domains architecture is based on top of DCE/RPC, which was subtly enhanced and augmented, for very good technical reasons such as adding support for UCS-2 Unicode strings, and called MSRPC. MSRPC and DCE/RPC are themselves the most incredibly powerful middleware technology that existed at the time, and are still arguably the best and also the most under-appreciated middleware technology in the world, having been used in projects such as running the distributed file server for the 1996 Olympics Web Site; and providing tens of thousands of distributed access points to the UK's 800 gigabytes (in the 1990s!) of National Insurance Database, and handling hundreds of thousands of queries per week (every single Employer in the United Kingdom).

At the time, the researchers into NT Domains were unaware of all this: they'd heard of DCE/RPC but believed it to be proprietary, despite a BSD-licensed freely available reference implementation of DCE/RPC 1.1 being ironically the basis of MSRPC.

A cursory examination of The Open Group's Technical Reference material led the researchers to conclude that TOG's morass of documentation was "so much horseshit", close examination of which would only slow them down in the process of network-reverse-engineering the MSRPC protocols. Thus, the future of the researchers was decided pretty much for the next two years: mind-numbing examination of literally thousands of network traces.

On reflection, it's difficult to determine whether ignoring TOG's detailed, comprehensive and amazing work was a good idea or not. Certainly, later, the DCE/RPC Technical Reference material proved absolutely invaluable, and the DCE 1.1 Reference Implementation turned up as FreeDCE some time later. Certainly, the floundering and meandering caused much hilarity at the number of security flaws and continual crashes of Microsoft's Flagship Operating System, many of which were reported, responsibly and confidentially over the ensuing months, to secure@microsoft.com with only a hint of cheekiness behind them.

It's questionable whether following the "proscribed" route, by using the DCE 1.1 Reference Implementation, would have resulted in so many security flaws detected, and, as a result, it's debatable as to whether the bun-fights inside Microsoft over this quotes blatant breach quotes of their quotes Intellectual Property Rights quotes would have come down on the side of "get rid of them, at all costs" instead of "leave them alone, they're useful".

After the initial contributions from Paul Ashton, Duncan Stansfield, Linus Nordberg and others, the fledgeling "Samba NTDOM" team began to take shape, with quiet contributions from Anders Blomdell, Luke Howard, Matthew Chapman, Jean-Francois Micouleau and others. Their chosen task was to map out the contents of Pandora's box that had been opened by the simple key of joining an NT Domain. Out the window was the "Old" IBM-inspired LANMAN protocols used by Windows 3.1 and Windows 95; In was approximately five hundred completely unknown functions, across well over twenty critical MSRPC services.

It would be six months before a tiny fraction of these were usefully implemented; two years before enough were implemented with confidence, and well over five years before Samba 3 achieved "stable" status with enough of these protocols to become a "mainstream player".

August 25th 1997: Microsoft announced extending DCOM beyond Windows

In a wonderful case of retrospective irony, a search on the Internet Archive for a TechNet article shows this: an announcement, three days before the "Welcome to the SAMBA Domain" announcement, that Microsoft had plans to extend DCOM onto Unix, Linux, Solaris and DEC platforms.

By 2004, Software AG had withdrawn their implementation of DCOM (which had been, unknowingly, freely available); COMSource, from The Open Group, was available for Solaris - full source code including the Microsoft MSRPC MIDL.EXE compiler and an implementation of a Windows Registry Service - for a mere $USD 30,000.

Coincidentally, by 2004, Wine had been forced to pretty much implement total binary and header-file interoperability with DCOM, anyway, leaving just the actual DCOM98.EXE service as the only remaining proprietary blob in the picture.

DCOM was the object-orientated wrapper on top of the "straight c" MSRPC layer, and it is through the use of the incredible technology that DCOM is that Microsoft was really able to utterly overwhelm what little competition there was. Only in retrospect is it possible to see how far advanced Microsoft's core strategic technology was ahead of everyone else, and Free Software, bless its little cotton socks, still really hasn't caught on, over ten years later. It didn't help that Olivetti Labs (bought by AT & T) released the first version of omniORB, in May 1997 (see AT & T Archive from 1999) because a Free Software release of CORBA resulted in CORBA becoming more dominant, and even adopted as part of Gnome's core infrastructure.

1998: NTLMSSPv2

By 1998, Luke Leighton was working for Internet Security Systems. Specific encryption schemes had been identified. It can be surmised that the NTLMSSPv2 protocol update was inspired by the initial network-reverse-engineering announcement of 1997, which showed to the world that the basis of Microsoft's Flagship Operating System, Windows NT, had an utterly borked 40-bit DES-based algorithm at its core. The only country to whom this really wasn't a problem was France, whose laws dictated that encryption, for authentication purposes, was illegal. (This turned out, later, to be extremely useful in reverse-engineering MSRPC services: install the "French" version of Windows NT from the MSDN and suddenly the previously encrypted MSRPC traffice was in-the-clear!).

After ninety days of research resulting in thirty lines of code, the workings of NTLMv2 and NTLMSSPv2 was established. It was quietly implemented, and quietly added to "The Alphabet Book", officially known as "DCE/RPC over SMB: Samba and NT Domains Internals".

On informing Microsoft privately of this news, a formal specification arrived by return, outlining the full details of the NTLMSSPv2 protocol. The subtext: NTLMv1 is so broken that it would be a really good idea if Samba had NTLMv2 added to it as soon as possible (unfortunately, main-line Samba was way behind Samba NTDOM and so it would be nearly four to five years later before Samba received full NTLMSSPv2 support in its MSRPC and SMB services, along with the NTLMSSPv2 128-bit NETLOGON Secure Channel).

Interestingly, protestations that the NTLMSSPv2 algorithm itself had a 64-bit bottleneck, despite its claimed 128-bit design, fell on deaf ears. Any such protestations were pretty irrelevant, anyway, as the design of NTLMSSPv2, like all of Microsoft's Core Technology enhancements, were a fait-accomplit development behind closed doors.

July 2000: Samba and Windows NT Security Interoperability

By July 2000, enough research had been done to provide Unix and NT Sysadmins with a high-level set of options where they could enable better security and interoperability between Samba and NT systems. This research was presented at Usenix's Lisa 2000 conference. A tool to suck the NT's SAM database over to a Samba Server were included. Hilarious gaffes in NT's SAM synchronisation which effectively exposed every single password of the entire NT Domain were a mere footnote, published as "Why You Should Upgrade To NT4 SP4 or NT5.

Also, a dig at the incredibly irresponsible attitude of key Microsoft Engineers is still, nearly a decade later, unheeded:

Hopefully, now that it is known that these systems are so insecure it will encourage the use of a more responsible and already-proven (time and time again) attitude to security. Namely that the only safe way to develop security measures is to publish their workings and let experts analyze it - before they are actually used in production environments.

The Alphabet Book had been published, with the choice of front cover image being particularly apt: a well-to-do gentleman helping himself to a well-off lady's pearl necklaces. It this book that many of the Proprietary CIFS vendors bought at least three copies, in the hope that it would help the author. The irony is that only 650 copies were actually sold before New Riders went out of business, and copies of the book are now available second-hand at 50 to 120% more than their retail value.

(side-note: can anyone explain why Amazon's sponsored links include "Refurbished DCE Unimaster" fumigation units?)

2000 - The Samba NTDOM Architecture grows legs

At some random point it became clear that to continue with a monolithic smbd process containing all of the MSRPC services would be crazy. The Samba TNG Man Pages outline the daemons implemented, with each MSRPC service implemented in its own application.

The TNG Daemon architecture brought interesting challenges, such as:

• requiring NT security context information to be transferred securely between unix daemons running as root.
• implementing an authenticated NamedPipe service, emulated on top of Unix Domain Sockets, including initially setting up the correct security context.
• preserving MSRPC "threading" security contexts, which are when individual threads within an MSRPC application perform or require a different authenticated security context from the rest of the threads in the same client or server, over the same DCE/RPC channel!
• getting smbd to not care about the data it is proxying, but acting as a go-between for the security context
• providing a short-cut for all the vast amounts of authentication being performed when the transport can reasonably be identified as "local" - ncalrpc.
• properly implementing NETLOGON SChannel which does recursive authentication in the case of Inter-Domain Trust.
• Ensuring that smb.conf reloads worked as best as possible. The parameters were subdivided into three groups: connection, authentication and service, where connection comprised IP address, NetBIOS and DNS names; authentication introduced unix name and group, NT name and group macro substitution, and finally (the most contentious), connection to a service, which would always be IPC$ in the case of an MSRPC service, would result in the loss of continuity for smb.conf macro "includes".
• Treating much of the infrastructure as if it were "joined to its own domain", by adding a machine account password for the machine itself. Even over loopback (on ncalrpc), this machine account would be utilised to securely authenticate the MSRPC services, thus preserving security, and also greatly simplifying the design of the TNG architecture by not forcing "loopback" communication down a separate special-case code path. Thus, a simple test "is the name dot or backslash backslash dot" if so use ncalrpc otherwise use ncacn_np would be all that was required to distinguish emulation of an NT workstation from emulation of a Primary Domain Controller!

Interestingly, it would be seven years before the Samba Team would begin to realise that they needed to implement a daemon architecture, and, despite rejecting the TNG architecture out-of-hand at the time, the Samba "Franky" architecture has the exact same design principles as Samba TNG, as it faces the exact same issues that were dealt with successfully, back in 2000.

In the intervening years, Samba would continue to dominate these critical ports and services, disallowing by design (as well as free software license incompatibility) any chance for other projects to contribute or interoperate.

A good analogy would be for the Linux Kernel developers to dictate to the Apache2 project that the Apache Server must be compiled and linked into the Linux kernel (not as a userspace service, but as a kernel-space driver), and, by inference, released as part of the Linux source code base and also, by being part of the Linux kernel, released under a GPL license.

Clearly that would be insane to expect, and would not in the least bit be tolerated if suggested, but that's exactly the situation, with regards to the NamedPipes transport, SMB IPC$ transport and the NetBIOS transports in Samba: Samba's monolithic design utterly dominates and excludes all options but those dictated by the Samba Team, and has done for over a decade.

2001: The Linuxcare Dot Com

By 2001, with the Dot Com bubble still flexing its muscles, Linuxcare had funding of $USD 100 million, and was looking to become the next Redhat or the next VALinux massive IPO gamble. Numerous big names such as Rusty Russell and Andrew Tridgell had been brought on-board, and it was on Andrew's recommendation that Luke Leighton was added to the Linuxcare Australia team, with a view to bringing main-line Samba finally up-to-speed with the research that was in the "NTDOM" branch.

The "NTDOM" branch was a fork of mainline Samba, back in 1998, and the fork point from which NTDOM took off was only after about eight months of research. As a result, Samba 2.0 had a very limited MSRPC implementation that remained stagnant for several years. This stagnation caused significant problems for Samba, and resulted in wasted contributions re-developing work that had already been successfully implemented and proven to work in NTDOM and TNG.

Linuxcare represented a badly-needed opportunity for Samba to "catch up".

November 2001: Winbind, the Single-Signon Holy Grail

The combination of Andrew's systems-level expertise and enthusiasm, Tim Potter's steady programming and Luke Leighton's long-standing research resulted in an outstanding achievement: Winbind. At last, it was possible to have a Unix system that could be part of a Windows NT Domain, with account names and account passwords stored in either a Microsoft proprietary Windows NT SAM Database, or even in a Samba Server's emulated SAM database! NSSwitch and PAM plugins merged NT Accounts and NT Passwords respectively into a Unix world, at last.

The advantages of using Samba and Winbind even in a pure Unix / Linux environment should be clear: tens or hundreds of thousands of accounts can be stored in a manageable secure database, subdivided by "Domains" if required, with Inter-Domain Trust established between departments, and, if necessary, and required, Windows NT Primary Domain Controllers brought into the mix.

With Samba and Winbind, even the Unix / Linux workstations are secured, as the workstations need to be added to the Samba-controlled Domain (in the same way that NIS+ operated).

The Linuxcare Bubble Bursts...

Linuxcare's treatment of Luke Leighton was almost criminally negligent. Internet Security Systems had been absolutely outstanding, with an H1-B1 visa obtained within months, and interim employment arranged with professionalism. So the precedent had been established, and the procedures worked out. These successfully-deployed procedures were relayed to Linuxcare's management team, with the expectation that they would be put into effect.

The warning signs had already emerged, when, even before joining, Linuxcare neglected to pay Luke Leighton's travel expenses to a CIFS Interoperability conference. Within three months, and outgoing costs of approximately £3500 per month with no pay cheques yet received or even discussed, Luke Leighton was 10,000 miles from home, £12,000 overdrawn, sitting in a hotel at 4am wondering what the **** was going on, and extremely distressed.

The effect that this situation had on relations with Andrew Tridgell were "not good", to put it mildly. It also didn't help that the entire Linuxcare Australia team, with the exception of Luke Leighton, were allowed to own their Intellectual Property, having all joined before the new Investors took control. Luke Leighton made a specific conscious commitment to terminate development of any source code and any further Samba-related research, until the Intellectual Property issue was resolved satistfactorily (which it wasn't). Fortunately, with over two years of research material to explain to Andrew, the decision not to develop further Intellectual Property was never going to an issue.

However, what was an issue was not the lack of new material, but the sheer quantity of old material. Along with the effects of the negligence of the Linuxcare management (which was eventually resolved - almost grudgingly and far too late), the transfer of knowledge to Andrew didn't go too well. It became clear that Andrew was both out of his depth, with respect to the comprehensive nature of the NT Domains protocols and the mismatch between the NT approach and Andrew's significant Unix systems-level knowledge, but also Andrew was becoming increasingly distressed by the effects that Linuxcare's negligence was having on Luke Leighton, as well.

By the time that Linuxcare began to address the issues (such as paying Luke Leighton regularly and adequately), the damage had long been done. Fortunately, the Winbind infrastructure is the lasting legacy of the Linuxcare fiasco: at least there was one positive contribution to the advancement of Free Software that came out of that awful time.

Insights, Losses and Responsibilities

Many years later, it can be seen that carrying out empirical observations to derive expert knowledge is a skill which leaves very little room for "exactitude and certainty". Offering "concrete proof" and "concrete evidence" is utterly impossible to do when the knowledge has been derived through literally thousands of manual, laborious test cases, many of which are long-since forgotten. When presenting such highly-derived information to people whose skills are specifically in the arena of Unix File Sharing, Unix systems programming, Unix Security, it can be seen that it would be really quite easy for such people to dismiss empirically-derived knowledge that appears to be so "etheric" and "flimsy".

It would also be incredibly easy to utilise expert Unix systems knowledge as a weapon to terminate arguments, and, when the subject being discussed is actually Windows NT Internal architecture, concrete and proven Unix knowledge clearly wins outright against complex conjecture and vastly stacked logical deduction.

In the end, then, it is users who have suffered most by this mismatch. More specifically, it is application developers who have been let down so badly, because users and Systems Administrators do not care about how MSRPC, DCOM. SMB and NetBIOS work, they care about how to use the services. In that regard, Samba continues to serve its users well; the Samba Team fulfil their implicit duty and their responsibilities to Free Software, and they deserve the utmost respect for doing so, for well over a decade.

Continuing with the documentation of advances in "Catching up" with Microsoft's NT Domains-based protocols, the subject of how developers are the ones that have been let down by the Samba Team will be discussed later, after careful review of further developments.

April 17th 2001 - Andrew Tridgell joins Jeremy Allison at VALinux

Samba Leaders join NAS Team

June 27th 2001 - VALinux drops NAS hardware (and the Samba Team)

ouch. The Dot Com bubble starts to implode. Jeremy Allison goes to work for HP; Andrew goes to work for Quantum (another NAS company).

Implications of HP's interest in Samba

Here's where something goes slightly awry with Samba's development. HP's interest in hiring Samba Team members is to ensure that the MSRPC spoolss service, and associated infrastructure, is working correctly (for the benefit of HP's Printing Division). With very little underlying infrastructure, the manually-created MSRPC layer puts a very large strain on the Samba infrastructure, and a considerable drain on the resources of the Samba team, for several years.

Jean-Francois Micouleau's initial work on spoolss for Samba NTDOM, had taken him nearly eighteen months to implement one of the biggest and most comprehensive of the NT Domains MSRPC Services: a whopping seventy functions. all by hand-coded marshalling (without an IDL compiler or even an IDL definition file - completely from on-the-wire network reverse-engineering). Jean-Francois' work on spoolss in Samba NTDOM alone expands the NTDOM source code by something like 30%. Even Wine's implementation of spoolss covers eleven DLLs.

At a crucial time, having strived so hard to ensure that control of Samba's development remained in their hands and their hands only, the Samba Team leaders were now being asked to incorporate infrastructure that they didn't fully understand. It would be nearly four years before this was achieved, even with a proven and successful implementation available to them, in the form of Samba TNG.

2002 - Luke Howard develops XAD: Active Directory for Unix

XAD was Luke Howard's implementation of Active Directory. Years before Samba's decision to reimplement Kerberos and LDAP, one person took FreeDCE, Heimdal, OpenLDAP and Samba, produced a few patches, implemented a PAULA Daemon, implemented proprietary SAMR, LSA and NETLOGON services (thanks to FreeDCE's BSD license), and released an interoperable Active Directory Server Product for Unix.

XAD Installation Instructions

It would be several years before Samba catches up with Luke Howard's outstanding achievements.

March 2003 - Samba 4 begins

Samba 4 begins development, with plans to introduce an MSRPC-interoperable MIDL compiler, and to reimplement much of Samba 3, all of MSRPC, all of the NT Domains MSRPC services, a DCOM server, and implementations of the Kerberos and LDAP protocols as well. (Note: Each of the free software projects that implement those protocols are at least a quarter of a million lines of code each, with multi-man-decades of specialist expertise and work gone into them, and the effort gone into the NT Domains Services measured in man-centuries of time spent).

No mention of interoperability for developers, at the Win32 level, is mentioned. At all. No mention of cooperation or interoperability for other projects (such as Wine) is made at this stage. Later on, Samba / Wine Conferences are arranged, as many people notice the similarity between the efforts of the Wine team and those of the Samba team.

The key focus of the Samba 4 development effort is on replacing the Samba 3 services, which are described as "showing their age". (Being based on code from 1998, at a time when very little was understood of MSRPC and its implications, that's an understatement of the first order).

25th May 2004 - Samba TNG ported to Win32 for use in ReactOS

In a Wine Interview with Steven Edwards, it's mentioned that Samba TNG has been ported to Win32, for deployment under ReactOS, the Free Software implementation of the Windows NT Operating System. This inspired Luke Leighton some time later to port FreeDCE and Samba TNG's hand-crafted MSRPC infrastructure to Win32, with a view to achieving the amusing goal of providing an entirely free software implementation of Microsoft's Flagship Operating System, as a Primary Domain Controller.

Aug 23rd 2004 - Advances in Samba 4

Advances in Samba 4 article is released. The most significant of the advances is the mention of the fledgeling reimplementation of MSRPC (despite the existence of a proven BSD-licensed reference implementation, FreeDCE). To date, Samba 4's pidl compiler still imposes a significant burden onto its developers, and leaves developers who would like to port Win32 DCOM or MSRPC applications to Free Software with such a significant maintenance nightmare on their hands.

In the paper, Volker also shows the significant age of the Samba 3 SAMR infrastructure, which was cut over from Samba NTDOM as far back as 1998, and remained pretty much stagnant ever since. In Samba TNG, the hand-marshalling of MSRPC was separated very early on from the actual "implementation":

RPC_SERVER_STUB_DECLARE(api_samr_del_groupmem)
{
    SAMR_Q_DEL_GROUPMEM q_e;
    SAMR_R_DEL_GROUPMEM r_e;

     ZERO_STRUCT(q_e);
    ZERO_STRUCT(r_e);
    
    if (!samr_io_q_del_groupmem("", &q_e, data, 0))
    {
        return False;
    }

     r_e.status = _samr_del_groupmem(&q_e.pol, q_e.rid);

     return samr_io_r_del_groupmem("", &r_e, rdata, 0);
}   

 RPC_IO_DECLARE(samr_io_q_del_groupmem, SAMR_Q_DEL_GROUPMEM, q_e)
{
    RPC_MARSHALLER_INTRO(q_e);

     RPC_MARSH_SUBCALL(smb_io_pol_hnd, q_e, pol);
    prs_uint32("rid", ps, depth, &q_e->rid));

     return True;
}

 RPC_IO_DECLARE(samr_io_r_del_groupmem, SAMR_R_DEL_GROUPMEM, r_u)
{
    RPC_MARSHALLER_INTRO(r_u);

     prs_uint32("status", ps, depth, &r_u->status));

     return True;
}

These are the hand-marshalled equivalent of what MIDL.EXE and WIDL.EXE auto-generate from the IDL file, resulting in sam_s.c and sam_c.c which are the server-side marshalling and client-side marshalling, respectively.

This leave the "actual" implementation, _samr_del_groupmem(), in a separate file, and in Samba TNG there are no less than three separate SAM database daemons to choose from, thanks to this design: an LDAP samrd, an smbpasswd samrd and a TDB-based samrd. Here is the implementation of samr_del_groupmem in the "unix-like" samrd:

uint32 _samr_del_groupmem(const POLICY_HND * pol, uint32 rid)
{
    DOM_SID group_sid;
    uint32 group_rid;
    fstring group_sid_str;
    DOMAIN_GRP *grp;
        struct sam_passwd *sam_pass;

     /* find the policy handle.  open a policy on it. */
    if (!get_policy_samr_sid(get_global_hnd_cache(), pol, &group_sid))
    {
        return NT_STATUS_INVALID_HANDLE;
    }

     sid_to_string(group_sid_str, &group_sid);
    sid_split_rid(&group_sid, &group_rid);

     become_root();
        sam_pass = getsam21pwrid(rid);
        unbecome_root();

         if (! sam_pass)
        {
                return NT_STATUS_NO_SUCH_USER;
        }


     DEBUG(10, ("sid is %s\n", group_sid_str));

     if (!sid_equal(&group_sid, &global_sam_sid))
    {
        return NT_STATUS_NO_SUCH_GROUP;
    }
    DEBUG(10, ("lookup on Domain SID\n"));

     grp = getgrouprid(group_rid, NULL, NULL );

     if (!grp)
    {
        return NT_STATUS_NO_SUCH_GROUP;
    }

     if (!del_unix_groupmem(grp->name, sam_pass->nt_name))
        {
                return NT_STATUS_ACCESS_DENIED;
        }

     if (!del_group_member(group_rid, rid))
    {
        return NT_STATUS_ACCESS_DENIED;
    }

     return NT_STATUS_NOPROBLEMO;
}

The declaration of this function can clearly be seen to be completely compatible with the IDL file definition outlined in Volker's paper:

/************************/
/* Function    0x18     */
NTSTATUS samr_DeleteGroupMember(
        [in,ref]                 policy_handle *handle,
        [in]                     uint32 rid
        );

This is just one example, but it can be clearly seen that the plan for Samba TNG was always to migrate to an MSRPC interoperable compiler, as the entire suite of Samba TNG MSRPC services also have the hand-marshalled NDR infrastructure separated out from the actual implementation of the service, and the client-side hand-coded functions implemented as closely as possible to conform to the same IDL.

5th January 2005 - Andrew Bartlett's Samba 4 Active Directory Paper released

Samba 4 - Andrew Bartlett's Active Directory Thesis outlines how Samba 4 will provide Active Directory Server functionality. Also documented is the NTLM v1 and v2 authentication; how NTLMSSP is used in MSRPC and other areas; how Kerberos works and how Microsoft overloaded the PAC and much more.

It's interesting to note that, after Andrew Bartlet worked with Luke Leighton, with guidance from Luke Leighton on how NTLMSSP works in Samba TNG two years earlier, out of 64 references in the Thesis not one mentions Luke Leighton, in an area where he is an acknowledged expert. Not even in section in 10.1, which covers the History of Samba, is it mentioned that the Samba 2 DC functionality was from an early cut-over of Samba NTDOM work, from 1998. This in itself is quite an achievement. It's almost as if there was a desire to exorcise his existence from history.

7 Jan 2005 - Luke Leighton adds NT Authentication and ncacn_np (Named Pipes) to FreeDCE

A patch to FreeDCE is released which brings FreeDCE up-to-date and MSRPC on-wire interoperable. Integration with Samba TNG is added, hooking in to its NamedPipes emulation layer, allowing FreeDCE to have an "ncacn_np" transport. Support for the MSRPC "3-way-authentication" handshake is added to FreeDCE's authentication infrastructure.

20th Jan 2005 - Luke Leighton mentions NamedPipe interoperability on Wine lists

In a precursor to the Samba "Franky" architecture, Luke Leighton describes how ImpersonateNamedPipeClient could be implemented on top of a NamedPipes emulation layer.

In particular, the solution to "inheriting" security contexts over NamedPipes is described, by extending MSRPC to include a new PDU packet type. "Pre-Authentication" of a sorts, if you will, which of course can only be carried out over cast-iron-securable transports (such as root-only-accessible unix domain sockets).

By extending MSRPC in this way, it's possible for projects that use and implement DCE/RPC to interoperate in a "clean" manner, without having to worry about Free Software License incompatibilies. Any project that has an MSRPC-compatible IDL compiler can use the IDL file, containing the security context, as the specification.

No actual code need be exchanged or cross-licensed.

February 2005 - Luke Leighton Reverse-engineers MAPI and Exchange 5.5

Luke Leighton carries out MAPI and Exchange Exploration, sufficient to get Outlook to open an Inbox. IDL files are created for MAPI, emsabp and emsmdb. With the emsabp and emsmdb implemented in NDR, FreeDCE easily made mincemeat of them. MAPI, being a proprietary son-of-a-whore RPC format, required a different approach, and "aparser" - the precursor to pidl - was adapted and press-ganged into service.

An OpenChange developer picks up the developments, ports them to the current Samba 3 infrastructure, and, some time later, Samba Team members take notice and begin work on creating an OpenChange MAPI library.

Up until that exploratory work, no progress had been made on Exchange Interoperability: the OpenChange project had stalled, and had focussed on back-end JET Engine reverse-engineering. After the exploratory work, OpenChange is up-and-running, and is now a part of the Samba XP Conferences.

The primary focus of OpenChange is unfortunately on a MAPI client, not an Exchange Server replacement. Luke Leighton's and Matthew Chapman's early reverse-engineering accomplishments showed that progress is made much faster when the client, the server and the network tracing are all tackled at the same time: they each bootstrap each other forward and allow for better understanding and, crucially, a better test matrix framework.

22nd July 2005 - Wine DCOM success

Wine DCOM success.

Wine announces some progress in the implementation of DCOM interoperability. This interoperability is utterly necessary for key services and applications to work, in Wine. Missing from the Wine DCOM "picture" is the actual DCOM service (implemented in for example DCOM95.EXE and DCOM98.EXE). The rest of the Wine DCOM infrastructure is, incredibly, pretty much complete! Win32 DCOM Applications can be compiled as-is.

2nd February 2005 - Samba's DCOM progress

Implementing Distributed COM in Samba.

Jelmer outlines the progress made in implementing DCOM in Samba 4. Examination of the source code shows that it does not contain an implementation of DCOM "pinging", but it may contain enough for the Wine team to start replacing DCOM98.EXE.

As MSRPC binary-interoperability was never part of Samba 4's roadmap, let alone header-file interoperability, it's impossible for Win32 developers to utilise Samba 4's DCOM infrastructure for serious consideration of porting Win32 / DCOM applications to Unix.

It will be necessary to wait until the Samba "Franky" NamedPipes emulation layer is added to Wine, and also it will be necessary to wait until the Wine Team examine Jelmer's valuable efforts, and port them to Wine's much better MSRPC and DCOM infrastructure.

26 Jul 2005 - Samba TNG announces Embedded PowerPC success

Samba TNG announces a successful port to embedded systems. The internal daemon architecture allows non-essential MSRPC services to be completely removed with ease, dramatically reducing the size.

16 Sep 2005 - Samba TNG MSRPC Services successfully ported to Win32

An announcement is made indicating that rpcclient.exe is possible to compile for Win32, as is smbclient.exe, and also lsarpc.exe. binary downloads are made available, including nmbd.exe!

The msrpc-client.c commit log puts this as around 16th September 2005.

The significance of these endeavours is that ReactOS stands a chance to have SMB and MSRPC network interoperability. A native SMB Server for Win32 is essential: it's simply impractical to expect ReactOS to have "Services for Unix" and then run a unix-compiled Samba under that POSIX emulation layer, or worse, expect ReactOS to run the cygwin user-space POSIX emulation layer: the performance penalty and the known instability issues of cygwin would be atrocious.

20 Dec 2005 - Announcement of progress in porting FreeDCE to Win32

As part of providing ReactOS and Wine with a reference implementation of DCE/RPC (augmented to be MSRPC-compatible), FreeDCE was ported to Win32. Later, echoservice.exe and echoclient.exe were successfully created and announced.

Binaries available for download of rpcd.exe (the DCE/RPC Endpoint Mapper!), the pthread-win32 dll (pthreadGC2.dll), echoservice.exe and echoclient.exe.

April 2006 - Wine / Samba Conference highlights need for NamedPipes

Wine's Named Pipes Samba Interoperability Requirements Specification

Rob Shearer documents the discussions and notes from the Wine / Samba "cooperation meetings". No actual work is carried out until, two years later, Samba itself requires the exact same interoperability interface (yes - the exact same interface that had been successfully implemented in Samba TNG, seven years earlier).

2006 - Novell acquires PADL's XAD Technology (and announces it 2 years later)

Padl OpenEnterp Announcement.

Wonderful Novell. gotta love them.

2nd September 2007 - Linux MAPI

Dan Shearer and Julien Kerihuel present Linux MAPI Programming over ExchangeRPC at the 2007 LinuxConf. The significant progress on integration of the MAPI client into numerous Free Software GUI front-ends is described, along with the limited progress on the Free Software Exchange Server replacement. Overall it's an outstanding achievement, at last.

2nd September 2007 - Wine Samba Networking Interoperability

Kai Blin's Wine Samba Integration talk describes some of the options for integrating Wine and Samba. This is the early precursor to the "Franky" Design.

D-Bus is mentioned in this early talk as one of the options. (Note: D-Bus is a bastardised version of all the best features of DCE/RPC shrivelled up into a pathetic attempt at inter-process communication. A comparison of the D-Bus specification and The Open Group DCE/RPC specification shows direct concept-for-concept equivalence, with the unfortunate fact being that the D-Bus specification only bothers to cover about 20% of the ground that DCE/RPC covers. Of particular concern is the lack of an IDL compiler in D-Bus, which is one of the absolutely vital features of any IPC mechanism, without which you might as well simply use 1,500 lines of c-code to transfer data between processes using unix domain sockets. Conclusion: D-Bus should have been shot at birth).

Sep 6th 2007 - Samba 4 Active Directory Interoperability announced

A Samba 4 AD article explains the status of Samba 4's Alpha 1 Active Directory Server interoperability release, with a warning that it is in no way ready for production use.

May 8th 2008 - Samba "Franky" Design

Samba Franky a combination of the best parts of Samba 3 and Samba 4. The architecture, utilising Unix Domain Sockets, is identical, concept-for-concept, to the Samba TNG Daemons architecture that had been implemented nine years previously.

It's ironic that internal interoperability requirements finally forced the Samba Team to make the right decision that will truly "Open Windows to a Wider World", not the Samba TNG proven architecture, nor the discussions in November 2004, nor the requests by the Wine Team to open up the protocols, in 2006.

April 17th 2008 - OpenChange Server progress

OpenChange Samba XP 2008 talk outlines the progress made with OpenChange Exchange Interoperability. The MAPI Client library is good: the OpenChange Server, utilising Samba 4 infrastructure, is less functional.

Dec 2008 - Novel announces Active Directory Interoperability

Press Release

After purchasing XAD from Luke Howard nearly two years previously, and three years after entering into a License with the Devil that cost them one of their key strategic assets (Jeremy Allison), Novell announces that they have Active Directory Interoperability.

30th Jan 2009 - Kai Blin mentions Samba "Franky" Design on Wine-Devel

In a NamedPipes thread, Kai Blin mentions that Samba "Franky" at long last has the beginnings of the infrastructure that will make it possible for other developers and other projects to interoperate and communicate, thanks to a "Named Pipes emulation layer" inside forked versions of Samba 3 and Samba 4.

Later in the thread, Luke Leighton mentions how pleased he is that finally Samba looks like it's actually going to deliver on its promise of "Opening Windows to a Wider World", and offers some guidance from his experience nine years previously in this exact same specific technical area.

5th Feb 2009 - Luke Leighton ports "Com in Plain C" to winegcc

As an illustration to developers of how easy it is to develop DCOM applications utilising an entirely Free Software toolchain, Luke Leighton ported COM in Plain C, a well-respected guide to writing DCOM applications in c, to Wine.

The purpose of illustrating how easy and simple this is was to highlight just how far Free Software, and Wine, has come, and to illustrate exactly why the Samba Team has been holding things up with their intransigence over implementing NamedPipes.

Recap

Looking back on the developments since that initial "Welcome to the SAMBA Domain" announcement is fascinating. Due to lack of funding, and thanks to Luke Leighton's reputation as a leading Free Software developer having been destroyed, specific and limited strategic contributions are instead made which have a profound, indirect, hated, despised, entirely unwelcome but ultimately and eventually really grudgingly - and utterly privately - accepted influence on the direction of the Samba Team.

The most successful of these quiet strategic contributions has to be the MAPI exploration. Thanks to that unknown developer's reimplementation, the Samba Team had no idea as to where the work had come from, and so ran with it, with enthusiasm. The reaction of one of the Samba Team members, on discovering in 2007 the provenance of the work they were doing, was really quite violent.

So it took many years for Samba 3 to catch up to where Samba TNG had already successfully gone, with Samba TNG regularly being examined as a reference guide to understand NTLMSSP, NTLMv2, the NETLOGON SChannel, the internal workings of the key strategic MSRPC Services such as SAMR, NETLOGON and LSARPC, and much more.

The benefits of the internal architecture of TNG as separate daemons for each MSRPC service was clear as daylight, with the benefits and possibility for swapping out individual services on a case-by-case basis and replacing them with, for example, Wine-derived services with a real purpose behind them, being blindingly obvious. Wine already has three really important services that could clearly bring significant benefits to Free Software if turned into proper networked MSRPC services: SVCCTL (for starting and stopping services); SPOOLSS (for printing) and WINREG (the Wine Registry Service).

Even the Samba "Franky" design will benefit enormously from such an architecture, as the best components from all of the projects in this field - FreeDCE itself, Wine, Samba 3, Samba 4 and Samba TNG - can be picked not by the Samba Team but by any developer or systems administrator. Inter-project cooperation can only be a good thing; encouraging cross-project testing across such a well-designed and well-defined interface such as DCE/RPC will only result in significant improvements to each (as the CIFS Interops already prove).

The addition of Kerberos and LDAP into the mix is almost a minor hiccup - a footnote or a speedbump taken at 60mph - and as Luke Howard's patches showed, once you have SAMR, NETLOGON and LSARPC client and server infrastructure, extending Heimdal and OpenLDAP and then adding PAULAD is pretty straightforward. Even the undocumented PAC format isn't hard, as it must comprise everything that a NET_USER_INFO_3 structure already contains.

The lack of support for DCOM is a significant failure, even on my part, for which I can only whole-heartedly apologise to all the people who have been let down, by having to pay extortionate Microsoft Licensing fees, and for having no choice but to utilise proprietary compiler technology and libraries. Unfortunately, this is what happens when you don't financially reward the right people - the ones who quietly get on with the job: they have to take up work as common labourers, or earn money doing web sites, instead of helping break the back of a billion-dollar monopoly to give you back your freedom.

I have one really significant apology to make, and it's this: that I did not have the strength at the time to overcome the egoistic bullying that I was subjected to. The consequences alone of Samba not having a NamedPipes emulation layer are beyond measure, financially, but run into billions of dollars of your money, forcibly spent on proprietary technology because there was no alternative. I made an attempt in November 2004 to re-engage with the Samba Team, to impress on them the absolute paramount strategic importance of a NamedPipes emulation layer. After a simple acknowledgement from Jeremy Allison that it was a good idea, I let you all down by getting very distressed at the amount of time that had elapsed since i was last forced into silence when previously discussing the same technical issue, having been informed in no uncertain terms that it was a useless idea that would never work and would never be securely implemented, especially by someone who had been told in clear unequivocable terms was technically incompetent at Unix systems programming.

Well, once again, Samba TNG is proven to be the correct technical approach. Nine years late, Samba "Franky" has been forced to implement a NamedPipes emulation layer, utilising unix domain sockets, transferring NT Security Contexts over them, and providing the full semantics of NT's NamedPipes. Quite literally exactly as I had implemented, in 2000.

Once again, the Samba Team has been shown the way, by an unpaid, unfunded and much disrespected Free Software advocate who keeps coming back, despite people's belief that he is insufferably arrogant. The definition of arrogance is a combination of ignorance and ego. Where I know I am not knowledgeable, far from wishing to hide my ignorance behind an egoistic mask, I am overwhelmingly happy to be corrected (willing to learn so that I do not waste any further time in achieving my goal), willing to learn with eagerness and alarming speed. The rate at which I incorporate new knowledge has always been one of the things that has made it so difficult for people to accept me - especially those people who lack confidence, and my willingness to admit ignorance and be corrected was - and often still is - perceived as "weakness". And regarding ego: if I am egoistic, then why am I exposing myself to the continuous ridicule by re-raising these issues year on year? Surely, a truly egoistical individual, bent solely on scoring a few points against those much-respected leaders of a significant free software project, would have given up a long, long time ago?

No, the fact is this, that even after a decade of silence, I still have a detailed understanding of this scarily-large technological topic spanning millions of lines of code and literally man-centuries of development effort. I have a clear and unequivocal appreciation of how strategically important these protocols are, in ways that not even those who have presented evidence to the E.U Commission and the U.S. Dept of Justice fully appreciate. Consequently, I have a duty and a responsibility wayyy beyond that of most people that you will ever encounter in your life, to ensure that those people who have pushed me aside for their own personal reasons and their own personal gain, and thus have taken over in part my responsibilities to you, are hounded until their absolute last dying breath to ensure that they actually fulfil the responsibilities that they have assumed.

And those responsibilities do not just include delivery of File and Print Services for Unix, as is believed to be so important: that responsibility has been achieved a long time ago, and the Samba Team deserve our utmost respect and praise for delivering and maintaining such services. No - there is much more: also included is responsibilities and a duty to bring to developers the freedom from proprietary toolchains such as MSVC and the Microsoft Developer Network, in this little-understood, obscure, esoteric and incredibly powerful technical area. It is in this area that both I and the Samba Team are letting you down so badly, for which i genuinely, genuinely apologise.

At the top of the chain of key strategic Microsoft protocols is DCOM, with pretty much everything underneath that single protocol spanning millions of lines of code and dozens of different services. When it is possible for an entirely unencumbered free software toolchain to compile and run a DCOM application - better yet, when it is possible to compile even Python-DCOM with an entirely unencumbered free software toolchain, running under an entirely free software infrastructure on the platform of your choice, and for that application to interoperate seamlessly with other Win32-based DCOM applications, then my responsibility and duty to you will have been fulfilled.

I trust and hope that that will be well before 2019.

Lastly, I have to reiterate that my praise of the Samba Team for finally beginning the implementation of a NamedPipes architecture is truly genuine, because a NamedPipes layer truly is the way to "Open Windows to a Wider World".