This document is a general description of the WOAD system. However, it is written not as a tutorial but as a working design sketch. Hence, the exposition follows the design process and includes the designer's musings and the rationale for major design decisions.
WOAD stands for Web-Organized Application Development (see here for alternative meanings) -- a web-based application development tehnique, philosophy, and toolset (WOAD Warrior) based on browsing, annotating, and editing the application's source tree using a web browser and specialized codebase server. Putting it another way, the idea is to make an application (or collection of applications) into a website around which the developer community is organized.
Note that this technique is orthogonal, but complementary, to the idea of a ``Web Application'' -- WOAD can be used to organize any application, though it is highly optimized for web applications (e.g., it's easy to go from a web page's source to a browser preview). It is also possible to link a web application with its WOAD server in order to make a tight coupling between a user community and a developer community.
WOAD can be thought of as a kind of ``IDE for the Web'' as well as being a
browser-based IDE. The general idea is to build a web-based tree browser,
code viewer (with syntax-based color highlighting, as in Emacs
font-lock-mode), and documentation extractor (e.g. JavaDoc),
with full cross-linking between keywords and documentation and between
application web page previews and the corresponding source code. In
addition, any page in the application, and any file in the source tree,
can be collectively annotated, just like the manual at php.net/. It's sort of like Code Warrior meets
SourceForge in a Wiki-Wiki Web.
It's actually slightly more accurate to regard WOAD as a Dis-integrated Development Environment -- it works quite happily alongside whatever editor, version-control system, build manager, web server, upload program, and so on you are presently using in your development.
WOAD can be used in several different ways, depending on whether it is a personal tool or a shared environment, and whether it is separate from or integrated with the application being developed:
warrior in the PIA bin directory runs a
stand-alone version.
woad in the PIA bin directory runs a
WOAD server.
WOAD requires:
WOAD works well with but does not require:
make
emacs
The applications and development tools above are the ones that WOAD is designed to work with ``out of the box'' -- it should be easy to add interfaces to others.
The ``understanding'' part comes from the ability of the code viewer to link every identifier or keyword with its corresponding definition (if it has one) and documentation (if it has any). For example, in the statement:
String foo = new Table(bar.getSomething())
The idea would be to link String, a system class, directly to
its documentation at Sun, Table, a PIA class, to its local
source file (which in turn would link to its javadoc
documentation), and bar and getSomething to
their declarations.
Every file would be linked to, typically, two editors: one web-based and implemented directly in the WOAD server, and one user-specified application (e.g. Emacs). Suitably-equipped editors would have links back to the browser.
Similarly, every file would be viewable both in source form through WOAD, and in final form through a local copy of the target server. The latter would be visible through multiple browsers (e.g. Netscape and Lynx).
In the various WOAD views, there will of course be controls for building
derived files, handling version control, updating the website, and so on.
These will have to be configurable for different build programs
(make and ant), different version control
systems (CVS and RCS)), different update
techniques (ftp, rsync, scp, ...)
and so on.
It will be easy to attach annotation to the code, and in some cases to see it (or at least find it) from the resulting web pages. Linking between annotation and code, and vice versa (from the code view to the annotation) will be as simple as possible.
Some applications, of course, will not want to make annotation and developer-feedback links available on the final website, but some (e.g. php.net/) will, and WOAD will make this functionality available too, e.g. by means of PIAServlet.
Two general styles of annotation are required: threaded discussions, which are best for gathering input from a large user community, and collaborative ``anything goes'' web editing, which works best in a small developer community. Text formatting and linking conventions will be the same for both.
This is the simplest case. I start with a copy of WOAD Warrior on a local machine, and use it either by itself or, more likely, alongside my regular editor to develop a web application.
Another use for WOAD Warrior will be as a tree browser and customizer for imported applications. In other words, I download a source tree and set up a parallel WOAD tree for annotation.
In this mode, WOAD is just a personal tool along the lines of an IDE.
Once the web application is deployed (for example, on an Apache server with PHP), I have the option of deploying all or, more likely, part of my WOAD tree for PHP-style documentation that my users can add to. (See below.)
This is almost as simple. A development team sets up a CVS repository for a new application. Each team member then uses a personal copy of the WOAD server as a tree browser and editor, occasionally checking their changes back in.
In this case the source tree and the WOAD annotation tree are separate. Ordinary programming is decentralized using CVS, as usual, but everyone shares the same WOAD tree so that annotations can immediately be shared with all other developers.
Even in the centralized case, an individual developer may well run a local copy of WOAD Warrior as a code browser over their personal code tree, and to keep personal notes and personal customizations. The local copy will of course proxy to the shared one.
Once an application is deployed, there are suddenly two communities to worry about: the developer community and the user/customizer community. In this case there may be two distinct WOAD trees overlaid on the same source tree: one for the developers and one for the users. It's also likely that the users' view will be restricted; e.g. to only the web pages.
Because WOAD makes it easy to develop specialized, web-based editors for configuration files and web pages, it's a good way to deploy specialized customization tools for an application.
Customization is of course going to be done locally at each installation of the application, so there will be many customization communities springing up, each around its own WOAD server. Each group of users will have to decide on a policy about which annotations and customizations to share with other groups; the simple default is just to keep them local.
Even with a local WOAD server, requests that can't be fulfilled locally will be proxied up to the centralized server so that changes in the shared content will be immediately available. Note that there will need to be ways of locking both WOAD pages and application files so that users trying to customize things don't step on the originals.
There will be some kinds of applications that can be implemented directly over a WOAD server: mainly applications that include discussion forums, collaborative editing and annotation, and collaborative development of structured web pages. For example, dmoz, Wiki, and slashdot are examples.
A ``real-world'' WOAD-based application might be part of a larger website, perhaps an intranet.
There are actually four sets of URL's involved in a Woad-organized application:
It is vitally important that the target and annotation URL trees be isomorphic. This extends as far down as query strings: query strings in the target need to be available on the annotation pages, because annotations may apply to specific queries.
Note that POST requests present a significant problem here -- there is basically no way to recapture them in most cases. They can be recaptured if the target server is Woad-aware and has links to Woad on all of its pages, or if the Woad server is being used as a proxy.
Corresponding to the URL trees above, we have the following parameters
that define their relationships. Note that all of the following
parameters are expected to start with a ``/'' character and
not to end with one.
notesPrefix
The location of the root of the WOAD annotation tree relative to the root of the WOAD server. It is normally only needed if the WOAD tree is on the same server as the application. Note that we do not need to put this on the command line -- if needed, it's guaranteed to be somewhere in the target server's configuration.
notesPrefixdefaults to null. If a non-null value is required it should normally be ``.notes''.
woadPrefix
The location of the WOAD application (forms and documents) relative to the root of the WOAD server.
woadPrefixdefaults to ``.Woad'' and will rarely, if ever, have to be changed. In fact, it is presently fixed and changing it might be very difficult
docOffset / offset=
The location of the document source relative to the source root.
sourcePrefix / prefix=
The location of the source root relative to the WOAD server root.
If a target URL corresponds to a real source document, the latter
can be found at sourcePrefix+docOffset/...
Note that notesPrefix is not automatically
prepended to sourcePrefix -- this allows the source
tree and annotation trees to be at the same level.
sourcePrefixdefaults to ``.source''.
sourceSuffix
Source directories are overlayed by their
annotations. This makes it possible to distribute annotations
alongside sources. When it is necessary to annotate individual
source documents (files), the annotation for source file
.../f will be found in
.../sourceSuffix/f/.
sourceSuffixshould probably default to.notes. This is ambiguous ifsourcePrefixandnotesPrefixare both null.
wordPrefix / words= The location of the keyword map relative to the WOAD server root.
(Presently undefined.) If a keyword has a map entry, it can be found
under wordMap
wordPrefixdefaults to ``.words''.
Prefix - prepended to a target path to make a WOAD path,
or the location of a WOAD tree component relative to the WOAD root.
Suffix - appended to a WOAD path to reach a related
component.
Offset - location of one part of the WOAD tree relative to
another.
Keywords are a bit of a problem. One option is to make them all queries,
another is to put them under /.Woad/Words; we don't do either
of these, choosing instead to add another prefix, so that the keywords can
all be found under /.words.
This is not strictly true at the implementation level: the directory/.Woad/Wordsdoes exist, and is simply aliased as/.words-- this makes it possible to include some pre-built headings and indices with the distribution.
Keyword pages can get fairly complex, because a word may be used in several contexts, and these will want separate annotation. This suggests that each word should correspond to a directory. This wins, because it makes it possible to associated arbitrary amounts of information with a word, and there's minimal interference between the different items.
The directories immediately underneath /.words correspond to
contexts in which words can be defined. For example,
/.words/func contains words declared as functions;
/.words/file contains words used as the names of files.
It will probably prove necessary to hash words into subdirectories on their first character (or maybe even their first two characters).
Similarly, it sounds as though the best thing to do for annotation may well be to make a directory for every page being annotated. This makes it easy to separate annotation files from ``real'' target pages in a listing: the real pages and real subdirectories map into directories; the annotation maps into files. It also makes it trivial to obtain a specific piece of annotation: simply tack its name onto the URL.
We should handle missing documents by passing the requested URL and pathinfo in a query to the listing file in the last valid directory.
Source annotation is a different problem. Making every source file
correspond to an annotation directory would be severe overkill, and would
make it impossible to distribute annotation together with sources. Hence,
we will put source annotation in subdirectories (with the name
sourceSuffix) of the source tree.
Note that things are still fairly simple: the source tree is a virtual subtree of the WOAD root, so the annotation will simply hang off the corresponding real subtree. Since the source tree retains its structure, it will be trivial to map filenames into URL's, and relative links in source files will still (usually) work.
It will be possible to use WOAD as a ``pure'' source browser or directory
browser: this simply corresponds to the situation where
sourcePrefix is null, and there is no target server and no
corresponding target annotation tree. Putting the target URL annotations
further down in the tree, e.g. hanging off the source document root, is
another possibility (for the future), as is putting source and URL
annotations on separate servers.
The above implies that the current work is by no means wasted: the situation in the source tree is essentially the same as it is now, only at an offset from the main (target URL annotation) tree.
make, emacs, ...
Note that this is only one of many possible orderings: most of the ``steps'' above are independent.
In most cases it will be necessary to keep WOAD configuration files and
so on carefully separate from the application's own files. In particular,
we have to stay away from things like HEADER.html,
_subsite.xcf, *.xh, Makefile, and
so on. This suggests:
_woad.xcf config file
WOAD/ annotation directory
*.woad XHTML files
*.ww (Woad Web) annotation files
woad.ts tagset
There are actually several good ways of doing the annotation; we'll probably want to be able to handle all of them:
WOAD/ or .woad/ subdirectory and a
_woad.xcf config file in every source-tree directory.
This is most suitable for an application developed from the ground up
using WOAD.
WOAD/
subdirectories just to keep it out of the way.
.woad files in every source directory. This
is most suitable for an application which is basically nothing
but a WOAD tree used as a collaborative authoring and discussion
environment. Even then it would be simplest to keep annotation in
subdirectories for security reasons, so we'll ignore this case for now.
Possibly the most difficult decision will be where to put WOAD's own source tree -- or even whether it should have one!
WOAD annotation directories. Parsers, tag
handlers, etc. would be scattered around the PIA tree as appropriate.
After consultation with Greg it looks like the best choice is PIA/Apps/Woad, so here we are.
The flip side of ``where it lives'' is how you navigate around a WOAD site. My guess is that all of the alternatives below will be wanted, in different situations.
/home/httpd, config files in
/etc/httpd, and source code scattered around all over the
place.
cgi-bin or servlets) under the
virtual root. This probably works best for straight web site
development.
path/WOAD or
/WOAD/path, gets you from an application path to
the corresponding WOAD directory. This is even a more difficult
mapping than looking at the document root, but is exactly what you want
for previewing. This is clearly the optimum for a collaborative
application (for example an intranet or an open-source project like the
PIA) that exposes its own WOAD tree.
There are three different cases for a project:
The conclusion is that, in all cases, a project page is just the index page of the project's WOAD tree. This simplifies things.
So, now that we know what a WOAD site is supposed to look like,
how do we structure PIA/Apps/Woad and the PIA
root directory (~/.pia) to make it look like that?
There's a bit of a problem, in that Woad can run in several different configurations:
/../, so it may be necessary to
use, for example, /Woad/ (possibly something like
/--/ or /.Woad/ if we want to make it less
obvious).
/WOAD/.
There are also four plausible configurations; they ought to be selectable on a per-project basis:
Getting the real root to point to the source tree may be tricky. It
requires either rewriting the _subsite.xcf in Warrior's real
root directory (and rereading it, which isn't supported yet), or adding
some kind of mount(path) method to
Subsite. Getting the virtual root to point to the source
should be easier: just create the new real directory and write its
_subsite.xcf file. (But does this really work? It's
possible that the _subsite.xcf file gets read as soon as the
directory is created, at which point we're in trouble.) It's
probably simplest just to fix Subsite so that it can reread
the config file -- we need to do that anyway.
During initial implementation we can finesse this problem by hand-building the necessary project directories. And we can build it into the top-levelwoadscript for starting the server. It's not a problem for theThisPIAproject, either, because we can just put the necessary stuff intoPIA/.
The big problem here is that we want to be able to view files in multiple
formats. This means that the most obvious thing, just redefining all the
tagsets, doesn't work. What we really would like is an automatic
wrapper document that does an <include> of the file
with the appropriate tagset, based on a query string.
It might be possible to dance around this problem, too, in one of the following almost equally disgusting ways.
<define wrapper> perhaps. This is
elegant and almost clean, but it means that you can't change
the viewer's tagset based on the query string or current state.
Combined with one of the other schemes, though, it could give you a
good default view.
Apps/Woad/viewer.xh/path --
is the code in place yet for passing what a CGI would call
PATHINFO down to the viewer? No: it ends up giving
a ``404: not found'' error. A trailing slash is accepted,
however, so there's hope. This scheme would work best, of course, if
the project was the viewer.
/- to an ordinary document's name, it lists
the directory it's in. This might be another, more elegant, way to go
from a path to a wrapper -- unconditionally appending is a lot simpler
than checking extensions and rewriting parts of a path. The Subsite
could look up the MIME type and default tagset and pass them down to
the viewer as entities.
All of these schemes get into a certain amount of trouble with images -- sometimes you want them wrapped, but sometimes you want to just see the darned thing. So this requires explicit rewriting in the wrapper.
All schemes have to worry about paths that start with
``/'' -- in other words, ``absolute'' paths that need to be
prepended with the server root.
Schemes that modify the path to splice in the viewer, rather than changing
the tagset mapping, make it much easier to get previews and view images --
just don't modify the path. The ``trailing /-'' scheme
additionally allows the extension map to specify the real tagset
for the page, which is a good thing, but it also means that the
tree is ``live,'' which may be a very bad thing.
My guess is that the right thing to do is the following:
<html> to fake start
and end wrappers.
&content; evaluation at
the same time to make it work efficiently.
/- to a wrapper for directory
listing.
/- mechanism or the pathinfo
mechanism (viewer.xh/path to simplify views.
The presence of the PIA project wants to be conditional on whether PIA_HOME is the virtual root. Some explicit checking might be in order to see whether one should make an Apache project.