Older blog entries for cdent (starting at number 136)

urls.map is a collection of patterns against which incoming HTTP requests are matched to dispatch those requests to the correct code. If you are familiar with Ruby on Rails it is similar to the routes file.

TiddlyWeb includes a default urls.map that has extensive comments explaining each mapping. You can see that file.

You may replace the entire urls.map with a different one by setting urls_map in tiddlywebconfig.py.

You may add to the urls.map or change how an existing mapping is handled with plugins.

selector is the WSGI application used within tiddlyweb to dispatch web requests to specific handler code. It is configured by the urls.map or via plugins.

selector was written by Luke Arno. It is a nice piece of work: simple yet powerful, focusing on one specific task and doing that task well. There is a web page for selector.

In REST a resource is an entity which is made available by the system, perhaps in a variety of representations. In HTTP, resources are identified and accessed by URIs. In TiddlyWeb there are four single entity resources (tiddler, bag, recipe and user) and several resources which represent collections of those entities.

On the web a representation is one of possible many forms of a single resource. In TiddlyWeb a tiddler is an example of a resource. It can be represented in a variety of forms including HTML, text and JSON. Each of these are a representation of the tiddler resource. A serializer is used to turn a resource into a particular representation.

A credentials extractor is a system in TiddlyWeb, part of the Auth Model that looks at an incoming web request and extracts, where possible, user information from the request and validates it. If the information is valid it is put into tiddlyweb.usersign for later use. If no information is found, the user is set as GUEST.

Multiple extractors are managed by UserExtract and configured by the extractors configuration item.

TiddlyWeb comes pre-packaged with two extractors:

• http_basic: Checks the request for HTTP Basic Authorization credentials and verifies them against the TiddlyWeb User datastore. Note: there is no challenger for HTTP Basic. It is assumed that if someone wants to use HTTP Basic they will just put the necessary information in the headers of their request.
• simple_cookie: Checks the request for a cookie named tiddlyweb_user with a hashed value. The cookie_form challenger will set this cookie to the provided username if the challenger passes.

While there is no requirement that an extractor do so, both above will query the User datastore with the discovered username to look for roles. If any are found they are added to tiddlyweb.usersign.

Additional extractors can be added to the system by adding module names to the extractors list in tiddlywebconfig.py. The modules should contain a class called Extractor that implements the ExtractorInterface.

An instance is a single collection of TiddlyWeb data, the store, with a (potentially empty) configuration. There can be any number of instances on any machine.

In most cases an instance is manifest in the filesystem as a directory containing the store, tiddlywebconfig.py and any custom plugins used in the instance.

Instances are created in two ways:

• You intentionally create a new instance with an instancer tool, such as twinstance from TiddlyWebWiki. See Create an Instance for an example.
• TiddlyWeb automatically creates an instance for you because you have run a command requiring one:
  • Calling twanager server will start a server on the default port of 8080. The curent directory becomes an instance if it is not already.
  • Running any twanager command which uses the store (e.g. adduser, bag) will cause the current directory to become an instance if it is not already.

Wrote up Request Overview, describing the steps for a single request for a single tiddler.

My server, the one that hosts tiddlyweb.com and hoster.peermore.com went down again and it's taking some time to get it back up again. This is annoying.

One way to understand how TiddlyWeb operates is to examine what happens in the system when a GET request for a single tiddler is made. Since the primary role of TiddlyWeb is to make tiddlers available, this touches on the main features. As an overview this touches on, rather than describes, the main concepts. For more depth, follow the links.

In a default instance a tiddler may be requested via two URIs:

• by bag via /bags/{bag_name}/tiddlers/{tiddler_title}
• by recipe via /recipes/{recipe_name}/tiddlers/{tiddler_title}

To keep things (relatively) simple this description will address the bag-based request. This is appropriate because the canonical URI for a tiddler is the one which address the tiddler in its bag. For a description of how things are handled when addressing the recipe see recipe cascade.

TiddlyWeb is architected as a stack of WSGI applications wrapping the core tiddlyweb application. This stack is hosted by (or mounted on) a web server. See Architecture Overview for more information.

Assuming a bag name of cars and a tiddler title of buick a GET request proceeds as follows.

1. The web server receives the request, creates an initial WSGI environment and hands off to the first application in the stack.
2. Environator logs initial request information to the debugging log (if enabled).
3. Configurator adds the current tiddlyweb configuration to the WSGI environ.
4. Query parses and processes any query string parameters and any POSTed form parameters into tiddlyweb.query in the environment. Any parameters which are filters are parsed as such, and added to tiddlyweb.filters.
5. StoreSet establishes and initializes a connection to the store defines in configuration.
6. UserExtract runs the credentials extraction process. If an authentic user is found their information is added to tiddlyweb.usersign. If no user is found, the user is set to GUEST.
7. Header manipulates a HEAD request into a GET but truncates the output. For this GET request, nothing happens at this layer.
8. Negotiate determines what representation of the requested resource is desired. This happens in one of two ways: By processing the Accept header or by evaluating an extension provided at the end of the URI. The determine type is put into tiddlyweb.type for later reference.
9. The selector app examines the URI and determines which handler (defined in urls.map) maps to this request URI and method. In this case the mapping is GET tiddlyweb.web.handler.tiddler:get. That is, the get method in the module tiddlyweb.web.handler.tiddler.
10. That method looks at the name of the bag and the title and uses that to create tiddler and bag objects.
11. The bag is loaded from the store (established in tiddlyweb.store). If the bag does not exist an HTTP 404 is sent as a response.
12. The current user (in tiddlyweb.usersign) is checked against the read constraint of the bag's policy. If the user does not pass, a PermissionsError is raised, to be handled by the PermissionsExceptor (described below).
13. If the current user can read from the bag, the tiddler is loaded from the store. If the tiddler does not exist a 404 is sent.
14. The headers of the request are checked for cache-validation headers (If-None-Match or If-Modified-Since) and compared with tiddler data. If the cache is valid a 304 is sent as a response, with no body.
15. If there are no validation headers, or they are invalid, the tiddler is serialized to generate the output content.
16. Headers are set, a status of 200 is set, and the content returned to the rest of the stack.
17. If the output content is text/html and the requesting user-agent appears to be a browser, the HTMLPresenter is engaged to wrap the HTML in a useful presentation. Note: HTMLPresenter is often disabled of overridden in plugins.
18. The PermissionsExceptor checks for policy violations earlier in the processing and turns them into appropriate responses. If the cars bag's policy required a user for access, the current request will be redirected to the challenger system.
19. The HTTPExceptor checks for HTTP exceptions raised earlier in the code and turns them into appropriate responses. If the cars bag or buick tiddler don't exists, a 404 exception would have been raised. It is here that it is turned into a well formed response sent to the user agent.
20. EncodeUTF8 run to translate unicode content into UTF-8 encoded byte strings. Internally TiddlyWeb keeps all textual content as unicode, transforming to/from UTF-8 at the boundaries.
21. TransformProtect adds a Cache-Control: no-transform header to protect against broken caches which transform content (such as minifying JavaScript).
22. SimpleLog logs the request to a log file (if configured).
23. The web server sends the headers and body to the user agent.

See Also:

• explorer
• bag
• policy
• constraint
• store
• serialization
• credentials extractor
• challenger
• resource
• representation

Did some refactoring in the TiddlyWeb code to clean up handling of content-length and content-type on a PUT or POST. Before the cleanup it was possible to generate a 500 (instead of a more correct 400) if a request was made without those headers.

Worked on the data integrity ideas mentioned in 20120327, updating the database on tiddlyspace.com and cleaning up the tiddlywebplugins.sqlalchemy3 code. Unfortunately the motivation for the work was to address issue 922 but the changes did not fix it. See the ticket for more info. The new code is more correct, though, so not a total loss.

Sometimes you may wish to create or update a tiddler, bag or recipe from the command line and twanager won't work for you for some reason. Maybe you are accessing a remote server. In these instances you can use curl or wget to make requests to the TiddlyWeb server.

Below are some examples. In each of these the argument to the -d switch is a JSON string, so the use of ' and " matter. Please see JSON bag, JSON recipe and JSON tiddler for important details on the structure of the JSON that must be sent.

Create or Update a Bag

      curl -X PUT -H 'Content-Type: application/json' -d '{"policy":{}}' http://0.0.0.0:8080/bags/foobar

Create or Update a Recipe

    curl -X PUT -H 'Content-Type: application/json' -d '{"recipe":[["system",""],["common",""]]}' \
        http://0.0.0.0:8080/recipes/default

Create or Update a Tiddler

    curl -X PUT -H 'Content-Type: application/json' -d '{"text":"hello"}' \
        http://0.0.0.0:8080/bags/foobar/tiddlers/monkey

If you wish to PUT a binary tiddler you can do the following, assuming the source file is named monkey.png:

    curl -X PUT -H 'Content-Type: image/png' --data-binary @monkey.png \
        http://0.0.0.0:8080/bags/foobar/tiddlers/monkey

See also: HTTP API