Migrating from TiddlyWiki to Markdown files

After many years of using TiddlyWiki to keep my notes and journal, both personal and work-related, I have decided to migrate to a more simple solution: markdown files.

What is TiddlyWiki?

TiddlyWiki 5 (aka, modern TW) was a significant step in its development. The same software supported two platforms: node.js and the browser. A single-file HTML TW could be generated using node.js allowing a myriad of customization options. The client-side part (i.e. HTML/JS) supports multiple savers, including the single-file itself or TW running as a backend in node.js. No doubt, TiddlyWiki is an unusual piece of code.

Solution design

My focus was on leveraging free and opensource tools to build a combination which led to a simple solution to convert tiddlers to markdown files. After reading the development documentation and evaluating several alternatives, including creating a new TW module, I decided to take advantage of TiddlyWiki’s available commands.

The following block diagram shows the steps, programs and intermediate files that make up the process.

Solution implementation

These are the steps involved in the whole migration process:

  1. Setup empty TiddlyWiki5
  2. Load tiddlers
  3. Render to HTML
  4. Rename tiddler files
  5. Convert to Markdown

Setup empty TiddlyWiki5

We initialize an empty edition TiddlyWiki and add support for markdown and TiddlyWiki’s old format.

$(NODEJS) $(TIDDLYWIKI_JS) $(WIKI_NAME) --init empty
$(NODEJS) $(ADD_PLUGIN_JS) $(TIDDLYWIKI_INFO) tiddlywiki/tw2parser
$(NODEJS) $(ADD_PLUGIN_JS) $(TIDDLYWIKI_INFO) tiddlywiki/markdown

Load tiddlers

We can use the load command to load the tiddlers from several sources into a TiddlyWiki. One of these sources is a single-file html file (classic or modern):

--load wiki.html

Render to HTML

TW5 has a powerful render command that allows to render individual tiddlers, filter its contents and save the results in the filesystem. In our case, we render tiddler content and metadata in separate files to facilitate the conversion process.

--render [!is[system]] [encodeuricomponent[]addsuffix[.html]] \
--render [!is[system]] [encodeuricomponent[]addsuffix[.meta]] \
    text/plain $$:/core/templates/tiddler-metadata

Important: Loading and rendering has to be done in the same command-line execution to avoid having to persist tiddlers to disk.

Rename tiddler files

I want my markdown file names to be as much compatible as possible with any filesystem. This is implemented in this simple Node script.

// Remove accents/diacritics
.normalize('NFD').replace(/[\u0300-\u036f]/g, "")
// Make lowercase
.toLowerCase()
// Convert separators to low line
.replace(/\s+/g, '_')
// Remove any non-safe character
.replace(/[^0-9a-zA-Z-._]/g, "");

Convert to Markdown

Metadata is prefixed at beginning of the Markdown files using a format equivalent to YFM (YAML Front Matter). Tiddlers in HTML are converted to CommonMark using pandoc (the universal document converter). This variant of Markdown was the one that produced more simple results in my tests.

It’s interesting to see how the GNU Make utility is used to convert all files using one rule:

$(MARKDOWN_DIR)/%.md : $(TW_OUTPUT_DIR)/%.html
    @echo "Generating markdown file '$(@F)'..."
    @echo "---" > "$@"
    @cat "$(^:html=meta)" >> "$@"
    @echo "---" >> "$@"
    @$(PANDOC) -f html-native_divs-native_spans -t commonmark \
        --wrap=preserve -o - "$^" >> "$@"

As html files don’t exist until this final step, make execution must be done in two separate commands:

$ make # or 'make export-html'
$ make convert

Conclusions

I am quite happy with the final result. This little project took me three full days, including evaluating other design options and learning a bit about the internals of TiddlyWiki (worth a look!). It’s amazing how using free and opensource software allows you to put together powerful applications with few lines of code.