Older blog entries for lkcl (starting at number 67)

well, life is certainly interesting. news. news. so much has happened since my last confession^H^H^H^H^H^H^H^Hdiary entry.

let's see. i went to ukuug in london, last month (or maybe more :) and met steven tweedie, alan cox, hans reiser all for the first time and an old debian... person, phil hands whom i have not met for a long while. i had a really good time (thanks, alastair, for organising ukuuk 2000).
it was really productive and enjoyable conference. i gave a pinhole-view onto some of the things been done that have resulted in the samba tng development. i described how we have been network-reverse-engineering the nt dce/rpc services, resulting in client and server implementations that end up with functionality to provide regedit and samedit as client-side programs, and winregd and samrd as server-side programs.
talking to steven, alan and hans was extremely beneficial: we discussed means to add things like Unicode support in ResierFS; alternative security-authentication mechanisms for file access; extended attributes in ext3 and ReiserFS, loads of stuff. Adding these things would make Samba's job easier, but they don't just benefit Samba.
Steve and Alan described to me how Coda and even the Linux NFS file services do call-backs from the kernel to user-space, for some of the file access / user-authentication requests. In this way, some out-of-band security data can be interpreted by a user-space service. putting several thousand lines of code - some of which may do TCP access, access files, malloc kbytes of memory, run functions up to a stack depth of about 15 to 20 functions, and generally cause mayhem and chaos - is TOTALLY inappropriate for putting in kernel.
Hans and i discussed putting Unicode in ReiserFS. As ReiserFS is object-orientated, he suggested adding "parallel directory entries". the idea is that you then have a conversion matrix between all the different formats, which you do _only_ once, and when you add / modify a file entry to / in a directory, you update each and every "parallel directory entry". so, if you add an entry with UTF8 to a directory, you call UTF8->Unicode and UTF8->unix-char* and UTF8->dos-8.3-mangled and then you call add_direntry_utf8, add_direntry_unicode, add_direntry_... etc. but you make this all object-orientated and generally less messy than this, but that's the principle :)
I overheard Hans, Steven and others discussing adding extended attributes to ext3 and others. if you do an opendir() on a file, you get a DIR* with which you can list the names of the extended attributes of the file. what they then proposed was that ACLs, e.g. POSIX acls, NT security descriptors and any _other_ security mechanism, could be implemented as an extended attribute. i mentioned that whilst ext-attrs are a good idea and so is the opendir() on a file, implementing security mechanisms as ext-attrs is not. the reason for this is that if a user has write-permission to a file, he presumably also has write-permission to the extended attributes. if he has write-permission to the extended attributes, and the ext-attributes include specifying who the _owner_ of the file is, then the user can, without owning the file in one security-context, modify the ext-attrs and thereby gain ownership of the file through the alternative security-context. This Is Bad.
So, the only thing i can think of is to either have "dot-extended-attributes", i.e. extended attributes that start with some character, e.g. ".", are root-only accessible, or to have a separate API for extended-attributes that are intended to be used for security-information.

then, i went on a bit of a break. i was still on-call, and went round not only a few friends in cambridge, but a couple companies in germany and cambridge, too. then, i went to seattle, for the Lisa NT 2000 conference. that was kinda fun. they had wireless ethernet cards available - 2.4gb 11mbps wha-heeey :) :) they worked immediately, too. put the damn card in, type pump, then ssh samba.org and it worked! :)

on to san fransisco. there, i did some samba tutorials, explaining basically the things i'd been up to at ukuug and lisa-nt2000 and more. the hands-on tutorial was half-and-half vmware and samba, because it turns out that getting everyone to the point where they had useful nt4 and w95 vmware sessions going took *two hours*, leaving a remaining two hours for samba installs, configs, self-demos, questions etc.

then, back to oz. after two weeks here, having been separated from my skates since march, i couldn't stand it any longer! and had to buy some skates. Roces Khuti's, and K2 Escape's.
it turns out that canberra's road users are extremely aggressive and stupid. they seem to think that speed skaters belong on the cycle paths (which are only 4ft wide) and pavements (which are only 2 to 3 ft wide and usually made up of concrete slabs). it _also_ turns out that the road rules of australia require skaters to "give way to pedestrians and cyclists". well, excuse me, but if doing 15-20mph (24-32 kph) how in hell's name am i supposed to "get out of the way of a pedestrian or cyclist" who decides to just... step into my path? in other words, it is *extremely* dangerous and also impractical to skate on cycle paths at those kinds of speeds [and that's on flat surfaces: speeds downhill on skates vary between 40mph (60kph) when there is only one person and 55mph (88kph) when in a group)]. also, a speed skater requires just over 4ft to skate in. that takes up pretty much the entire cycle path.
so, in general, it is either unsafe or impossible to skate on anything other than the roads. however, attempting to explain this in under one second to some FUCKING stupid canberra driver as they [as has happened once] attempt to force me off the road, exceeding the speed limit in doing so and breaking at least two other traffic rules, is simply not possible.

samba. samba development. my role in samba's development is at an end. i have been developing NT-compatible DCE/RPC services for use in Samba for the last three years. the decisions that i have been taking to implement those services have been based on both what i perceive, through black-box analysis of NT, to be as similar an implementation to NT as i can make them, and also to come up with and implement a mapping-scheme to the underlying operating system, which in this case happens to be POSIX-compilant Unix systems.
the reasons for doing this are quite straightforward:

• if it's been done that way in NT, then they probably thought about it quite a lot [some parts of NT _are_ actually very well thought-out and well-designed, usually the ones that are VMS security-based :) but there are several that are just as good]. duplicating this effort because of "Not Invented Here" syndrome, and what amounts to what i can only loosely encapsulate in "Unix Is God, Therefore I Can Do Better Than Microsoft And NT" syndrome is, if you ask me, counter-productive and a means to waste significant amounts of time and effort.
• working out How NT Does It means that you keep track of where NT is at, such that any _future_ improvements made to NT are easier to add to _your_ codebase.
• if you implement things in a similar way to NT, then you are likely to come across the same limitations and problems that the Microsoft engineers came across. You will _probably_ find that they actually _solved_ those problems. Trusting the set of services and their implementation methods is likely to save quite a lot of time because if you deviate from this, you are likely to _add_ to the problems and will likely have to revert to following Microsoft's footsteps _anyway_ as a reimplementation to solve the "additional" problems.

There is still plenty room for implementation freedom, design choice and coding improvements that make Samba a Better NT Than NT.

So, what happened? Well, since the beginning of the NT Domains for Unix project, in August 97, i ended up in a cvs branch. The reason for this was that the NT domains development was impacting on the release cycle too much. i tend to think things through for days, weeks or even months, then develop extremely rapidly, doing small cvs commits several times a day: the total number of cvs commit messages often exceeded 500 per month [we do have a lot of subdirectories, though].
depending on the type of work i was doing, i would either:

• do a preliminary, exploratory implementation, moving to prototype, then replace it with a first implementation based on that prototype, all in the space of under a few days or weeks. this approach i would typically use for a protocol area or service that is totally unknown, and the understanding of it subject to change minute by minute as more and more network traces are examined and understood. things like code quality, coding standards and safe programming practices, even things like freeing memory or even _using_ malloc and free instead of 256k stack-based arrays (!!! yes, the first implementations of samrd did struct USER_INFO[1024] where USER_INFO could be 512 bytes or more in size!), are a luxury and waste time in such a "development mode".
• do a second or third implementation based on the, by now reasonable, understanding of the service involved. the general aim of these reimplementations would be to "compartmentalise" the functionality being developed, such that it would be fairly easy to understand, stand-alone. to that end, i have been making extensive use of developing function-table-based APIs, and any other Unix and C programming techniques i come across that "divide and conquer" this mammoth task [Samba TNG's codebase is now 340,000 lines of code].

It is worthwhile making it clear that I develop code on the basis of future need, instead of solving only the problems faced and required _right now_. Future needs also include potential extensions and interoperability with other projects, whether those projects exist right now or they do not exist. I only do this if the boundaries are extremely well-defined and clear-cut, to the extent that even for Samba's own development it is worthwhile making use of those boundaries if only for modularisation purposes [breaking things down into simpler, more manageable tasks].
The only circumstances where I will *not* use such modularising and future-proofing techniques is if I am not aware of circumstances that warrant them!
For example, DCE/RPC over HTTP. I was not aware, until a year or so ago, that microsoft had added a means to provide [and proxy/tunnel through port 80] DCE/RPC over HTTP on port 689. I had given no consideration to this, so did not consider developing an API that allows DCE/RPC to go over different transports until that time. I was quite content to have the code that does DCE/RPC over SMB fully messed-up and mixed-up, with DCE/RPC-specific bits mixed in with SMB-specific bits.
Another example. DCE/RPC is capable of using different authentication methods [NTLMSSP is the typical one]. Service Pack 4 introduced a _new_ method for secure NETLOGON. i panicked: all my DCE/RPC authentication code was like, all of a sudden, a complete mess! the only authentication method was NTLMSSP, and now i had to support three methods: noauth, NTLMSSP and NETLOGON. So i created a client-side API and a server-side API which have five functions each, and there are now six files, {cli, srv}_{noauth, ntlmssp, netsec}_pipe.c in the Samba TNG source tree. cli_pipe.c looks up the auth requested in a table, obtaining a structure of function-pointers, one of which is called to create some state-information that is then passed in as a void* to the other functions called as-and-when required to do authentication / encryption encoding / decoding. And if it wasn't for Microsoft introducing NETLOGON, i would not have even bothered to code this up.

Needless to say, there are a number of things that conflict _really_ badly with the primary samba maintainers:

• Samba's development is samba-specific. Anyone who wants to interoperate with Samba's codebase can use the existing code as-is. The implication being that the burden of rresponsibility lies with the people who wish to leverage the services that samba provides is with them, not with the Samba developers.
• Furthermore, along the lines of Samba's development being samba-specific, any designs that extend beyond the scope of Samba-specific services, are not acceptable. This is presumably because the samba code-base is too large as it is, with the potential risk of Samba becoming responsible not only for Samba but for the other non-Samba-specific services as well. Any code that attempts to provide a clean break between Samba-specific and non-Samba-specific services is *also* not acceptable, because that is again "extra code" which must be difficult to maintain.
• Acceptable Samba coding does not include making provisions for future work, nor is it acceptable to implement designs that make future development easier. The only acceptable code is that which solves the known problems for known designs, known now. Any code that is not used or not part of the Samba-specific design is "cruft" and redundant, and should either not be written in the first place or removed immediately.
• Code of the highest quality and standard is acceptable. Code that compromises, even as a first and often only implementation, is not acceptable.

Some people may think that I am being somewhat unfair in my assessement, here. You may detect a bias in my description of the counter-arguments. Let me point it out to you. The bias is that I am describing not only the counter-arguments but also some points that are the beginnings of the basis for counter-counter-arguments. However, i note here, and make you aware that, that those are _my_ opinions, and _my_ biased, and now cynical, judgement.

Approximately one week ago, i put together, as i tend to do, the occurrences over a long period of time vis-a-vis my role in samba development, and concluded that my "allotted position" was, well... demeaning, is a good way to put it. I posted a message out to the samba lists saying that, basically, i quit. the end of the message said something along the lines of, if certain things are discussed and the outcome is acceptable, i don't quit. in other words, the message was a summary of the conclusions i had come to and also an opportunity for people to say, "hey, that would be really bad if you quit: your work _is_ valued - some of it - and we've been telling you that all along; exactly what are the problems, and what do you think can be done to work things out?". instead, the following happened [rough summary of approximately 80-100 email messages]:

• i received about 15 messages of support and "i wish you well", for which i will always be grateful.
• i received about 5 messages outlining either alternatives such as doing a code-fork or encouragements to reconsider quitting and work things out with the other team members.
• i received several messages from 3 of the samba team members with whom i have been working closely on the Samba TNG development, either requesting that i reconsider, do a code-fork or asking what projects i was going to be working on, and how could they help with those projects.
• i noted and/or received several messages which discuss the pros and cons of the different samba branches, their benefits, shortfalls and general discussion.
• i received several messages from the samba team members for whom my plans, ideas, approach, style and attitude to and for open source developments are either unacceptable, "too different" or "incompatible". these messages i took to construe as an "entrenchment" of the positions of these people. namely, that they simply dug themselves further into their positions that they have established over the last few years.

regarding the last point, in more detail. firstly, that some of those positions are probably valid, or more to the point, *acceptable* to a lot of other people, but enough of them are _not_ acceptable to me for it to be untenable for me to continue to work with them. particularly as i saw a lot of those points as demeaning and belittling conditions to work under. that they were repeated to me _even though_ i pointed out that i considered them to be demeaning caused me to just flip out, lose my temper and publicly insult them, for what amounted to what i saw as an equal insult, except using more words. i immediately apologised for the insult, but not for the reasons that _caused_ me to insult them.
secondly, my message was construed as an attempt to "override" what are quite sensible and acceptable Open Source development practices. it took quite a long time, and quite a few messages and conversations, to make it clear that a) this was not what i was asking b) this was not what i was asking. [i.e. first identify the mistaken assumptions, _then_ categorically state that the assumptions were wrong, and categorically state that i support the _same view_, not the incorrectly-assumed-view]. in the end, most of the misunderstandings were cleared up.

to summarise. samba's development is best done without my involvement for the time being. samba's users need to make their needs and opinions clear - not me - as to whether the current situation vis-a-vis samba development is acceptable. i am reluctant to lead a code-fork because i know that it will be a lot of work and will continue to foster the "NIH" syndrome that i believe that some of the primary samba maintainers are pursuing. some of this is due to lack of understanding of the features that Samba TNG provides. some is due to lack of *acceptance* of the features tha Samba TNG provides. some is due to my inability to explain in either simple terms or words of one syllable only, the concepts involved in Samba TNG's architecture. i do not understand, and i intensely dislike, why tried and tested code is being discarded when there is no suitable alternative that provides the same level of functionality, regardless of the standard of, or the perceived standard of, that code [again, the NIH syndrome has a large role to play, here].
so, in summary, an attempt to draw out either a conciliation / compromise or at least a clearer insight to, or confirmation of, the motivations of the primary samba developers and whether i could continue to work with them, i ended up with a resolution that leaves little room for reconciliation. i make no apologies; i have only one regret: the consequences of, but not the resultant outcome from, my anger and outrage.

what else am i up to? i am now investigating exchange. exchange is MAPI over DCE/RPC over TCP or HTTP. i downloaded the FreeDCE 1.1 runtime library and environment, with a view to adding NTLMSSP authentication to it and then maybe adding the HTTP transport to it and using it to develop an alternative exchange client and server. the client-side API could then be used by the gnome evolution team or anyone else who wanted to write an ms-compatible, non-win32-based MAPI client or MAPI server.
... except that the FreeDCE 1.1 runtime library uses DCEthreads, which are an implementation of the POSIX threads, DRaft 4 standard. they use sigsetjmp and siglongjmp to implement Exception-handling, and thread-cancellation, and they emulate all this on _top_ of LinuxThreads. if you use glibc 2.1.3 and linuxthreads from these, you get an infinite-recursion problem and a coredump. if you use redhat 5.2, which uses glibc 2.0.7, _then_ you have no problem :)
oh dear. anyway, i reported the problem to bug-glibc@gnu.org, installed a redhat 5.2 vmware session and am now examining the dce 1.22 code to see how to add authentication methods to the 1.1 runtime environment.

which brings me up-to-date at about 5am today. it is almost time for me to sleep, according to my somewhat STILL messed-up sense of time: i have been taking between four and six hours to get to sleep over the last two weeks.

i read the entire Axis Trilogy by Sara Dunnett in four days. I read her book Threshold in a continuous eight hour reading frenzy. I read the first of her continuation of the Axis Trilogy, Sinner, a third of it after finishing Threshold and the rest by the afternoon of the following day. I have not read the other two in the trilogy because i ran out of money for the month. I left my front door key in the house and it cost AUD $80 for a locksmith, and i had AUD $90 to cover me until i could transfer some GBP to my USD account in the U.S. so that i can get AUD$ out with the visa card in hole-in-wall machines here in Canberra. AGH!

i have run out of sci-fi and fantasy authors to read, so decided to plan a novel, myself. i hope it will be funny, but also make people uncomfortable by putting them on the edge of their seats in describing things, and peoples' reactions to them, both the descriptions and the reactions being plausible, awkward, highly evocative, with enough basis in real-life-like situations for fiction and science-fiction readers to actually think about, and enjoy, what's going on with the characters.

i met a really nice girl. she had just returned from a break to get away from her ex-boyfriend. she concluded that she needed time on her own, and that it was a good idea to drop contact with pretty much everyone extraneous to her life, and that definitely included me. i have done the same thing myself on occasion, so aside from being sad and rather annoyed at them, respect their decision.

one of my colleagues turned up at 2am, having been called by security because the alarm had been switched off and i was still here. well, *duur*, of _course_ the alarm is off, i'm still here!!

i miss my friends in atlanta, especially the skaters, and cyndi.

i miss my friends in cambridge, especially hedda.

i miss my family in cambridge, manchester, london, essex, a field somewhere in the middle of nowhere, france and suffolk.

i miss my dog, bonny.

i miss my car [310hp - 0-60 in 5.4s, 70-0 in 65yards.]

luke.

i keep writing these diary entries, and then remove them because they would cause more problems than their writing is intended to convey.

my situation, for the last four months, is such that i am in an extremely weak position - one that i accepted - and i am now reliant on other people in order to sustain my current status. those people are in three different countries.

as might be expected, starting from a weak situation, a lot can - and is - going badly wrong.

that's all i can really say without causing further problems for myself or the people whom i am heavily reliant upon in the situation that i mistakenly accepted.

auto-generating an smb client library. going well. plan is to do a server too.

things looking better than usual. mostly ups instead of downs.

hey, elrond's here! hi elrond!

oh, and hi again, mum. rest easy, enjoy life.

nearly wrote an extremely damaging diary entry today.

got it! after talking with raph, who suggested using net_flow_max_flow() on a per-user basis, i coded this up, checking the node_in_degrees (accumulated number of certifications in to a user at and above a certain level) is greater than the number of certs required. entry from ChangeLog:

Sun May 14 02:37:15 PDT 2000 Luke Kenneth Casson Leighton
<lkcl@samba.org>

 	* got it!  requires:
	- a call to net_flow_max_flow on a per-user
	basis, with the user as the seed.

 	- return of node_in_degrees * node_flow
	because node_in_degrees  tells you how many 
	paths there are into the seed [the user]

 	consequence: if each user has flow in from
	other users at a level
	equal to or above their own (node_flow[level] == 1),
	and the number of paths 
	(node_in_degrees[level] >= min_certs_reqd[level)
	to them is greater or equal to the number required
	to be certified at that level

 	*AND*

 	overall, they are certified in the "general"
	trust metric at that level

 	*THEN*

 	the certification is accepted.

patch to do number of certifications required be greater than x. node_paths is incremented not only when node_flow is incremented but also when it is found that there is a path [any path] to the supersink. could be implemented as returning node_flow[i] + node_sink[i]. not entirely confident about whether this Does The Right Thing!

--- net_flow.old.c      Fri May 12 06:21:51 2000
+++ net_flow.c  Fri May 12 06:10:32 2000
@@ -13,6 +13,7 @@
   int *edge_flow;

    int *node_sink; /* 1 if there is flow from this node to
the supersink */
+  int *node_paths; /* number of paths to the node */
   int *node_flow; /* total flow through the node */
   int *node_in_degree;
   int *node_in_degree_max;
@@ -100,6 +101,7 @@


        g_free (priv->node_sink);
+      g_free (priv->node_paths);
       g_free (priv->node_flow);
       g_free (priv->node_in_degree);
       g_free (priv->node_in_degree_max);
@@ -488,6 +490,7 @@
   priv->edge_flow = g_new (int, n_edges);

    priv->node_sink = g_new (int, self->n_nodes);
+  priv->node_paths = g_new (int, self->n_nodes);
   priv->node_flow = g_new (int, self->n_nodes);
   priv->node_in_degree = g_new (int, self->n_nodes);
   priv->node_in_degree_max = g_new (int, self->n_nodes);
@@ -499,6 +502,7 @@
   for (i = 0; i < self->n_nodes; i++)
     {
       priv->node_sink[i] = 0;
+      priv->node_paths[i] = 0;
       priv->node_flow[i] = 0;
       priv->node_in_degree[i] = 0;
       priv->node_in_degree_max[i] = 16;
@@ -579,6 +583,7 @@
            {
              next = pred_list[ix];
              priv->node_flow[ix]++;
+             priv->node_paths[node]++;
              if (ix != next)
                {
                  /* find edge from next to ix and increment
flow */
@@ -663,6 +668,7 @@
 #endif

           priv->node_sink[node] = 1;
+         priv->node_paths[node]++;
          q_ptr = q_beg;
          while (q_ptr != 0)
            {
@@ -1011,8 +1017,8 @@
 {
   int *result;

 -  result = self->priv->node_sink;
-  self->priv->node_sink = NULL;
+  result = self->priv->node_paths;
+  self->priv->node_paths = NULL;
   return result;
 }

demo works well, have to put in descriptions of different cert types. really want to tie mod_virgule in with other script systems e.g php. possibly possible with apache 2.0, with chaining of modules....

this kinda excitin bein in midl of discushns, glad 2 c advgato takin off. [rsi bad 2day]