Older blog entries for lkcl (starting at number 282)

sqlobject views

slight flaw in that code example - i had to use this:

class SiteSearch(sqlobject.SQLObject):
    class sqlmeta:
        #lazyUpdate = True
        cacheValues = False
        _cacheValue = False
    count = sqlobject.IntCol(default=None)

def dropTable(cls, ifExists=False):

if ifExists and not cls._connection.tableExists(cls.sqlmeta.table): return

sql = "DROP VIEW %s" % (cls.sqlmeta.table,) cls._connection.query(sql)

dropTable=classmethod(dropTable)

def createTable(cls, ifNotExists=False, createJoinTables=True, createIndexes=True, applyConstraints=True, connection=None): conn = connection or cls._connection if ifNotExists and conn.tableExists(cls.sqlmeta.table): return

sql = cls.createTableSQL()

cls._connection.query(sql)

createTable=classmethod(createTable)

i've just encountered the most _horrendous_ sql query i've ever had to design - it even beats the multi-alias-join thing to turn a sparse-entry recordset into a variable-width 2D table (for a demographic search)

the reason why i've had to use VIEWs is because the query has a COUNT record in it, and so requires a GROUP BY. the GROUP BY makes it impossible to do sensible multi-alias-joins, and not even a HAVING clause will do the trick.

so i had to first create the VIEW, then do a multi-alias-join multiple times on the VIEW.

aaaagh!

arse.

my project, the social networking one, has ground to a halt - from, believe it or not, mental overexertion :) an ordinary social net site: fine, no problem. lots of people posting, some chat stuff, blah blaahhhh. boring.

the easy stuff we managed in a reasonable amount of time (5 weeks). user-login, forum, chat, profile, picture uploading.

now it comes to the hard part: tagging, making the tags useful, and then search on the tags.

with a fourth normalised form database (ultimate object-orientated design) it's all gone slightly crinkly. getting results out of such a database has to be done with JOINS on aliases of the same (object/table) obj_table AS column1, obj_table AS column2.

now imagine putting "please count the number of times a tag has been put onto any object" into 4th normalised form. i can't quite get my head round it. i will - eventually... just not... this... month!

i fully expect to be successful - i will just have to warp my tiny brain and wrap it around the problem several times before it all fits into place.

in the meantime i'm bouncing off the walls cos i really want to get this project completed!

does anyone know how to make this work?

it's a more advanced version of /sbin/udevsynthesize which on debian will trigger SIX HUNDRED events, which takes forever.

i'm working on depinit, and so have split udevsynthesize down into separate scripts - one for "essential" devices, one for networking, one for block devices, and another for non-essential tty devices.

each time the word "add" is shoved into /sys/class/something/something/uevent, udevd picks it up and shoves a symlink into /dev/.udev/queue/<thedevice>.

when the scripts and modprobes for that device are finally run, the symlink is removed (by udevd).

so, in shellscript language, in my (four) udevsynth scripts named udevsynth-tty, udevsynth-essential, udevsynth-block and udevsynth-net, i'm trying to ONLY "watch" and "wait" for those files (symlinks) that i did an "add" on, and to ignore all other symlinks.

the first problem that i encountered was that i need to call readlink on each of the things in the queue directory.

it's all got quite hairy!

#!/bin/sh -e

function get_queue() { list="`/usr/bin/find /dev/.udev/queue -ignore_readdir_race -type l -print0`" if [ "y$list" == "y" ] ; then queue=1; return; fi echo "`/usr/bin/find /dev/.udev/queue -ignore_readdir_race -type l -print0|xargs -0 -n1 readlink`" queue="`/usr/bin/find /dev/.udev/queue -ignore_readdir_race -type l -print0|xargs -0 -n1 readlink`" } function check_links() { get_queue if [ "y$queue" == "y1" ] ; then return 1; fi echo "$file" | grep -qF "$queue" }

# this is a list of /sys/class/*/(*/)uevent # and has to be dirname-stripped to work with # the find/xargs/grep trick, above.

file_list2="$first $default $last"

for file in $file_list; do [ "$file" ] || continue echo 'add' > "$file" || true done

sleep 1

for f in $file_list2; do [ "$file" ] || continue file=`dirname $f` y="0" while [ "y$y" == "y0" ] ; do check_links sleep 1 done done

the social net site i'm working on is slowly getting there. my friend richard has done a total redesign of the underlying (4th normalised form) database. it's a _truly_ object-orientated database - totally abusing postgresql to have complete flexibility over data. i dread to think what kinds of search queries will find horrendous bugs in postgresql.

i've done a sparse-array -> 2d-array query before now, for demographic searches, using mysql, and mysql completely xxxxed up. in order to do the search correctly, i had to ask the customer to apply boolean logic to their searches, so that instead of NOT a AND NOT b AND NOT c they do NOT (a or b or c).

the customer was naturally totally unimpressed.

4th normalised form means that you have an entry with pretty much nothing but an ID and a "type" field, then you have to do a sequence of JOINs attributes AS attributes_table_NNN to add more and more attributes - effectively constructing the rows of the table dynamically.

here's the bit where mysql went wrong, when i also did SELECT where attributes_table_1.value == 'hello' AND attributes_table_2.value == 5 etc. etc. mysql returned completely the wrong results.

anyway.

i have more faith in postgresql.

once we've got this code underway, and are happy with it, i think it should be a pretty easy task to convince richard to free-software-license it. it's _really_ useful code if you're into object-orientated insane levels of flexibility.

it's not in the slightest bit obvious that there's a database underneath it.

i continue to be impressed with mod_python+formencode+htmltmpl. i think i will try simpletal because i really could do with the benefits of the tags being "buried" into the html rather than huuuge words TMPL_XXX - but, butbutbut... simpletal had _better_ have compiled-templates, otherwise it's out the door even before it's begun.

28 Apr 2006 (updated 28 Apr 2006 at 22:05 UTC) »
depinit

love it. my cute fujitsu's laptop is running very successfully with it. i am so lucky.

i have a battery life, from a default debian/unstable system, of 6 hours.

i have a boot time of 20 seconds which means i am not interested in doing stuff like "suspend". okay, i am - i would love to be able to save the state of whatever my current development setup is. more specifically: i no longer feel the need to worry about a THREE MINUTE startup time (of my acer c112) and no longer feel obliged to keep this one permanently switched on.

see, on my c112, i have STACKS of extra services installed - and i have to edit the /etc/init.d/XXXXX scripts to put, at the top "exit 0" to disable them - or worse, remove the symlinks from /etc/rc*.d

with depinit, i can have /etc/depinit/default/depend specify the minimum startup requirements ("normal" user mode with apache, ssh, x-windows and postgresql - all of which start up pretty much simultaneously).

then i can have some additional dummy service ... err... example... i have a requirement occasionally to use my laptop to pxe/netboot other systems: i can call that... oh.... pxeserv. so i create /etc/depinit/pxeserv/depend with the words dhcpserv, nfsserv, atftpd etc. which are the dependent services i will need.

depctl -s pxeserv will then run those required services - when i NEED them. not when initscripts says i have to have them.

new web site

i'm absolutely delighted with the combination of ajax, mod_python, formbuilder, htmltmpl and sqlobject - it's like... a breath of fresh air that leaves you free to think "what" rather than "how".

the site is (yet another) social network site - but it's something that _i_ have, _i_ am doing, _i_ enjoy - and, here's the important bit - that because i will know how it work, inside-out, can extend it with ease.

there's nothing worse than having to work with somebody else's code and going "ugh" and "i would have definitely done that differently in order to do xyz in the future".

in other words, i want nobody to blame for screwing up but me :)

profiles, photos, messages, forum. the forum i love: because i already had messages, adding "forum" took about five hours. to complete functionality. create, view forums, view messages, post message.

also i decided to put in a little trick: some javascript that refreshes the latest views - so if posts arrive in the meantime (including your) you get to see them automatically.

here's the neat bit: it's done by refreshing _only_ the list of 10 last messages, via ajax techniques. it's a perfect candidate for doing real-time irc.

you know - having discovered this trick, i can't imagine ever writing a web site that doesn't use ajax, ever again.

poems

as part of (yet another) rather silly exchange, in which some rather silly person (yet again) asked me to "get a life", i decided to post a warning message, referencing a couple of my poems which i believed were appropriate.

i then forgot about it.

part of the poems site keeps a record of the number of views on your work: out of my 50 or so poems online on this site, the views varies from 3 to 20 - averaging around 12 or so. not any more!! i took a look and went "wtf??? that's a mistake: 116 views? 83 views??? where did _that_ come from?"

so.

some people _were_ paying attention. this is good, cos that guy really _was_ being very silly. it's so unnecessary to think that difficult technical and strategic decisions can be avoided consideration by attempting to insult people. unless he apologises, i do sincerely hope that nobody allows him to make important technical or strategic decisions: his judgements are clearly clouded by personal hate.

23 Apr 2006 (updated 28 Apr 2006 at 00:04 UTC) »

_excellent_, muhahahah. i decided to try depinit on my shiny new fujitsu p1510 with debian/unstable.

the boot time is... 20 seconds, including starting postgres, apache2, sshd and xorg.

muhahahah

22 Apr 2006 (updated 22 Apr 2006 at 17:50 UTC) »

fucking-piss!!!!! bought a new laptop (fujitsu p1510d) and because i hate windows, i didn't boot in to it - just PXE-boot-installed debian (from hands-free - http://hands.com/d-i).

of course, the radio-rom isn't enabled, and there's no way to enable it from linux, so i fucking have to install fucking windows FUCK i hate windows.

[one more hour of my life wasted on windows so far]

[update] awwww xxxxit some more - it's a switch on the right-hand-side of the unit. i just trashed my lovely near-perfect setup only to find that there wasn't a problem :)

god i hate windows :)

well, i like mod_python so much i had to write an article about the web technologies i'm using. hurrah.

20 Apr 2006 (updated 20 Apr 2006 at 17:56 UTC) »

well, i haven't had much that i can write about, because i've been working. but i recently had some time available to start a project, and i started out with zope (which is what i am developing with at work) and within three days made a decision to abandon zope and use mod_python instead.

conversion - of the 1500 lines of python+templates i'd written up until then - was very simple and very quick, by virtue of having stuck to some guidelines:

1) for god's sake don't use anything in dtml except dtml-if (which should be a true-false test only), dtml-else, dtml-var (which should be simple variables only) and dtml-in.

DO NOT, under ANY CIRCUMSTANCES, use dtml-call or any kind of complex mapping.

2) anything complex is done by a python function, which _explicitly_ calls a variable (an instance of an HTMLFile) manually passing in the REQUEST dictionary and some additional keywords via **kwargs or named parameters:

    contents of index_html_file.dtml:
< html >
< title > <dtml-var "title">
< body > stuff < / body >
< / html >

def index_html(self, REQUEST): """ comment to keep zope happy """

return self.index_dtml_file(self, REQUEST, title='Hello')

by following these simple rules, i was able to install python-htmltmpl, and quite literally do a global search/replace on dtml-in with TMPL_LOOP, dtml-var with TMPL_VAR - you get the idea.

i've since investigated SQLObject/SQLBuilder, formencode, mod_python and htmltmpl, and ... i can't imagine ever doing, by choice, web programming with anything else.

273 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!