Blogged Images
How I build a website
A free toolset I cooked up, driven by XQuery, to build rich, data-driven websites
I've been using a particular toolset for the last few years to wrangle rich, data-driven websites without the need to install databases or learn multiple languages. It's about time I shared it with the world (thanks astro John for the impetus).
In summary, it enables you to create a rich, data-driven website, without the infrastructure or skills of a database developer or php coder, and isolating you from a bunch of problems.
Existing problems with web publishing
The approach eliminates (at least) the following chronic problems with standard approaches...
- Content which ends up mixed with formatting (Dreamweaver syndrome)
- Once you want to change the formatting, or use a different tool it takes hours of copy paste to retrieve your content.
- Data which is formatted as content (Microsoft Office syndrome)
- Although the information in the document is rich data, it's not possible to query it or cross reference it, because it's not stored as data. For example, dates of events end up as text in a bullet point.
- Server-driven applications beyond the control of mortals (LAMP
syndrome)
- Your friend makes you a website relying on a server somewhere, where your data is stored in a database somewhere, and relies on a complex bridge of technologies which live in only one kind of server environment.
Dreamweaver, Office and LAMP are all liberty-reducing approaches to building a website. They are chronic in the sense that you think your website is healthy, until you try to do something different with it later. At that point you find most of your work has gone down a one-way street and you are f***ed.
I want to put propose a liberty enhancing approach in which...
- Your data is authored directly by you, in a form you can access anywhere
and re-use for anything with low effort.
- Currently this is as XML text, but with upcoming work, this will be supported by a cross-platform authoring tool to write XML.
- The data can be queried richly to filter it and process it as you want.
- If you want to grab 'all events happening on a Thursday in Barnsley' and generate a nice table from them, you shouldn't have to pay an expensive developer to do it.
- This relies on XQuery, a data filtering language for entities (XML elements) and properties (XML attributes), which stands in for a relational database engine.
- The whole thing can be sent as a self-contained zip file starting at less
than 5 meg, including the source data, the generation tools and the
generated website.
- Regenerating the website from the content is a simple double click on Linux and Mac.
- This toolset also works on Windows. I don't have a Windows machine to author and test it on. Someone else who's taken the king's shilling can replace the two-line script ./output/write.sh which generates the site with a ./output/write.bat file, or they can replace their OS with something more liberty-enhancing.
- Disclosure: It relies only on a Java being installed, but this is available on most machines, and can always be installed.
Getting started
To begin, download the zip, or browse the structure online. This code is copyright free, so use as you wish. I'll talk you through the structure through the rest of this page.
The example content is something I put together for a presentation of this website building technique to the WIDOH lecture programme which I kicked off for a time at BT Labs. It reflects the simple sort of relational data which people often need to wrangle. When you've got to the bottom of how it works, throw it away and replace with your own.
The Nitty Gritty
The ./output/ directory contains the website which you can browse to see the cross-referenced database-like pages which have been created from the data.
Source data, simple format, simple storage
The place to start understanding the approach is in by looking at ./data/data.xml. This is simply a text file, but can be treated as a relational database. Here are some example entries,
<person
id
=
"cefnhoile"
title
=
"Cefn Hoile"
desc
=
"Inventor, Engineer, Troublemaker"
>
<div>
<h2>
About Me
</h2>
<p>
I work in BT's Future Technologies Group, creating new
forms of programming languages for non-technical people.
</p>
<p>
Anyone local who likes my projects should
join Ipswich's
<a
href
=
"http://curiositycollective.org"
>
Curiosity Collective
</a>
who experiment with playful technology
for arts and sheer pleasure. That's the forum for cooking up a lot of
the
more ridiculous ideas, as well as working out how
to actually build them.
</p>
<p>
In my really spare time, I'm a qualified yacht skipper and
paragliding pilot
</p>
</div>
</person>
<project
id
=
"semaphoresms"
title
=
"Semaphore to SMS Gateway"
desc
=
"A machine vision system which allows you to use Semaphore
to type and send SMS messages"
presented
=
"2008-11-10"
>
<contributor
refid
=
"cefnhoile"
/>
<div>
<h2>
My header about this project
</h2>
<p>
A paragraph about this project.
</p>
<h2>
My header about this project
</h2>
<p>
A paragraph about this project.
</p>
</div>
<div>
<h2>
My second header about this project
</h2>
<p>
A paragraph about this project.
</p>
<h2>
My header about this project
</h2>
<p>
A paragraph about this project.
</p>
</div>
</project>
Template logic and data queries
Next, take a look at the code for creating the website ./data/design.xq. Essentially the whole website is written in 5 lines, as follows.
for
$
page
in
$
allpages
return
c:writepage
(
c:href
(
$
page),
$
page
/
div[
*
],
$
defaultmenu) ,
for
$
person
in
$
allpersons
order by
$
person
/@
title
return
c:writepage
(
c:href
(
$
person),(
c:writelabel
(
$
person),
<
div
><
h2
>
Projects
</
h2
>
{
c:writelist
(
$
allprojects[contributor
/@
refid
=$
person
/@
id])}
</
div
>
,
$
person
/
div[
*
]),
$
peoplemenu),
for
$
project
in
$
allprojects
return
c:writepage
(
c:href
(
$
project),(
c:writelabel
(
$
project),
<
div
><
h2
>
Contributors
</
h2
>
{
c:writelist
(
$
allpersons[
@
id
=$
project
/
contributor
/@
refid])}
</
div
>
,
$
project
/
div[
*
]),
$
projectmenu),
c:writepage
(
'list_persons.html'
,
<
div
>
{
c:writelist
(
$
allpersons)}
</
div
>
,
$
peoplemenu),
c:writepage
(
'list_projects.html'
, (
for
$
date
in
distinct-values
(
$
allprojects
/@
presented)
order by
xs:date
(
$
date) descending
return
<
div
><
h2
>
{
string
(
$
date)}
</
h2
><
p
>
{
c:writelist
(
$
allprojects[
@
presented
=$
date])}
</
p
></
div
>
),
$
projectmenu)
Most of the file is preamble and standard function defs which could be imported as a library (I have left them exposed for transparency and hackability). The actual website is essentially written in the last 5 lines. If you want to see this code nicely color-highlighted, which helps, copy-paste it as XQuery code into pastebin.com or load in OxygenXML (see later).
Under the hood
The ./lib/ directory contains all the software which the two-line ./output/write.sh script invokes when generating the files in ./output/. On Linux and Mac, ./output/write.sh and ./output/clean.sh can be invoked by double-clicking on them to delete and then re-create the website. No doubt you can do the same thing in Windows, (I have in the past with this toolset but I don't have a machine to author and test this on so don't want to issue broken code).
Getting started on your own website
Xquery takes some getting used to and there are at least two very common errors which new authors experience, but if you start by copying this whole folder, and just modify the data and design to your purposes in stages, you can in principle create any website with it.
You can see, for example, that the project elements are processed by the line which starts for $project in $allprojects ...
So if you want to work with your own data, which happen to be events instead, then remove all the project, page and person elements from the ./data/data.xml file, and add your own event elements.
Subsequently remove all the references to page, project and person elements in the ./data/design.xq file and add lines which do what you want instead with your event values.
Getting a page design
You'll note from the code examples above that there's a writepage() function which has a load of HTML magic in it which makes the page look good. How do you get this?
Because I'm not a designer, I've been using templates from OSWD, as per my own website at http://cefn.com - based on a design from the oswd designer Nautica.
Once you've found a design you like, you just copy-paste the html straight into the design.xq file in the proper place, then replace parts of it with dynamically generated content as per this example.
For the expert
XQuery is a satisfying blend both for querying existing relational data, and for generating new content from it without nasty language bridges between the two. That's by design, since XQuery is a superset of XPath (an XML querying language) which extends the XML joining and filtering syntax with function declarations, string generation operations and XML (e.g. XHTML) constructor literals.
The mysterious ./form/ directory is there if you wish, to maintain the strict RelaxNG schema by which the data.xml file is validated. This isn't critical, but editing in a validating editor helps to maintain consistency, and force yourself to follow your own rules (e.g. to make sure you've put a date in an element which needs it for it to show up in a list properly).
Until I finish my graphical editor, which should isolate the user from the details of XQuery and XML, I recommend the excellent OxygenXML editor. This is a validating editor for both coding XQuery and for schema driven XML and is reasonably priced for personal use, (though crazy prices for commercial use). It will warn you when there is an error in the data.xml file, and does auto-completion for XQuery.
Finally
Good luck. Contact me if you've any questions about how to use the system.
Latest | Search | Contact