Why Drupal sucks

I use "sucks" here as a terminus technicus meaning "doesn't work in the way desired or expected".

Drupal "sucks" because it gets upgraded every couple of months and you have to upgrade. Now that's fine if you are using it in a business context where there are constantly paid employees to do that. But on a free or academic site, once set up it will be only occasionally updated. And when that new release comes out it effectively advertises the flaws to all the hackers of the universe: "Hey guys! You can hack Drupal sites with this neat exploit if the Drupal version is current-version minus 1". And of course they do. So you have to upgrade, and pronto, but you can't afford to. So you get hacked. Then you have to strip down your entire server because you don't know what the hackers did. And that really sucks.

Now all this would be gone if only the Drupal people would provide an automatic or at least easy upgrade path for new releases. Instead you have to take your site offline, then back up your database (why? Is the new release buggy?), then download the new server core, delete all your old stuff except the sites directory – and don't forget to cut and paste all those arcane rewrite rules from the old to the new .htaccess file – then install the new server, copy back the old sites, run the update script, take it back online and hope and pray that all your modules still work. They might not, especially that one you can't do without, whose author didn't bother to patch either. All this takes about half an hour, and you have to do it every 2 months on every damn Drupal site you maintain. So why not automate it? Is it really so hard? And don't tell me drush does all that because it only automates a couple of the easiest steps, and then you also have to worry about whether drush worked. I notice that Drupal 8 doesn't have an automatic update feature either. So that's why I think Drupal sucks. You're better off writing your own CMS. At least that way no one will know how to hack it.

mod_rewrite: redirect to another directory with additional parameters

Mod_rewrite is one of the hardest pieces of apache to get right: the documentation is awful, there is no way to test it and the syntax is arcane. All I wanted to do was redirect URLs from one directory to another, copying the existing GET parameters and adding some more. But there are a number of tricks to get it right. So my original URLs look like this:

http://me.org/mycms/olddir?param1=foo

And I wanted to redirect it to:

http://me.org/mycms/newdir?param1=foo&param2=poo&param3=roo

An additional requirement was that I wanted to add the rule to .htaccess in the CMS-directory, not root, since that's where the rules applied. The following rule worked in the "mycms" directory:

  RewriteEngine on
  RewriteRule ^olddir$ /mycms/newdir?param2=poo&param3=roo  [L,R,QSA]
  ... (other rules)

Since I was using Drupal, the <IfModule... was already present in the existing .htaccess file. So all I needed to do was add the correct RewriteRule. To explain what I understand by this: "^" means a URL starting with "mydir" since we are already in mycms (that's where the .htaccess file is), and ending ($) there. So only parameters follow. Now redirect to /mycms/newdir. If you don't start with "/" it will prepend the entire server path. Next, add your desired parameters, and finally append the existing ones via the QSA flag, which means "query string append". The "R" flag is needed because this is a redirect. The "L" flag is needed because otherwise other rules might be applied and this is the "last" rule we need here. I'm not sure all this is correct, but it is simple and it works. So I'm sticking with it.

Passing parameters to javascript via drupal_add_js

Nowadays most Web programs contain a significant Javascript (or JQuery) component but content management systems are still run in PHP, Java, python etc. In the case of the popular CMS Drupal, that language is PHP, and it would be rather nice if we had a reliable mechanism for passing arguments to a javascript file so that information would be available as parameters to customise its functionality. Unfortunately there is no such mechanism. Various hacks have been proposed and used within Drupal itself, so one can implant something like <script src="myscript.js?arg1=foo&arg2=bar"></script> and expect it to work, with the aid of a script that locates the script element, strips out the arguments and then passes them in javascript to a javascript function within myscript.js. Here's an example function that does it:

function getArgs( scrName ) { var scripts = jQuery("script"); var params = new Object (); scripts.each( function(i) { var src = jQuery(this).attr("src"); if ( src != undefined && src.indexOf(scrName) != -1 ) { var qStr = src.replace(/^[^\?]+\??/,''); if ( qStr ) { var pairs = qStr.split(/[;&]/); for ( var i = 0; i < pairs.length; i++ ) { var keyVal = pairs[i].split('='); if ( ! keyVal || keyVal.length != 2 ) continue; var key = unescape( keyVal[0] ); var val = unescape( keyVal[1] ); val = val.replace(/\+/g, ' '); params[key] = val; } } return params; } }); return params; } /* main entry point - gets executed when the page is loaded */ jQuery(function(){ // DOM Ready - do your stuff var params = getArgs('tabs'); var tabs = new myobject(params['target'],params['docid']); });

Using drupal_add_js

Another technique is to install a javascript file before the page is fully loaded so it will get executed when loading is complete. For this purpose the function drupal_add_js comes into play. It has to be called within a hook_init or hook_preprocess_page function. But since parameters to javascript files are not really allowed, this function escapes any content like '&', '=' or '?' into %26, %3D, %3F, so turning a request for a javascript file into something that is simply not there.

A workaround I used successfully is to precede the call to drupal_add_js('myscript.js','file') with another call to drupal_add_js(...,'inline');. In this way I managed to store the parameters in browser local storage and retrieve them similarly. e.g. for the inline call I used: drupal_add_js("localStorage.setItem('module_params', 'docid=english/harpur/h080&target=tabs-content')",'inline'); And to retrieve these values a function similar to the one above works just fine:

/** * This reads the "arguments" to the javascript file */ function getArgs() { var params = new Object (); var module_params = localStorage.getItem('module_params'); if ( module_params != undefined && module_params.length>0 ) { var parts = module_params.split("&"); for ( var i=0;i<parts.length;i++ ) { var halves = parts[i].split("="); if ( halves.length==2 ) params[halves[0]] = halves[1]; } } return params; } /* main entry point - gets executed when the page is loaded */ jQuery(function(){ // DOM Ready - do your stuff var params = getArgs(); var tabs = new myobject(params['target'],params['docid']); });

Compressing an unsorted list of integers

You see a lot on offer for compressing sorted lists of integers, but not so much on the rather uncomfortable question of how to compress a list of unsorted numbers. So why would you want to do that? Surely all one needs to do is sort the list then compress it? The problem is that in many cases the order of the integers represents information that sorting would destroy. Here is my case in point.

Actual hit-positions in a search engine

In a search-engine you have to make a list of documents in which a term is found. Each document is assigned an identifier. So say we had a list of documents in which the word 'dog' occurs. It might be found in 7 documents: 0,1,1,2,2,3,4,6,11,21,21. We can use an algorithm like FastPFOR because the sorted list can be converted into a list of deltas, or the differences between successive entries. These will typically be much shorter numbers than the actual values. In my case an array of 100 document identifiers compressed down to just 13 integers. Cool. But what if you wanted to store the locations in those documents where the word 'dog' occurred as well? This would bloat the index considerably, since 'dog' might occur 100 times in a single document. I could sort and compress it the same way, but quite often it would be really short, maybe only one entry. Then the compression algorithm would actually increase the list size by three or four-fold, since the overhead for a compressed list adds a number of ints to the start, and – this is the real killer – one compressed list would be needed for each document the word was found in. So trying to compress it the conventional way would first, probably increase the overall index size, and second, you would need to maintain a lot of compressed lists.

The solution

Ideally, we would like just one list of word-positions for each word, just as we had one list of document identifiers for each word. But such a list would have to be unsorted, because if we sort it we would lose the information about which documents those positions refer to. Fortunately, most positions are quite small. They can't be greater then the length of the longest document the word is found in. Or if it was found in only a few documents, or always at the start, the values would be even smaller. Lets say that all the values are less than some amount like 127. (Convenient, huh?) Then the list could be stored in 8-bit integers with one 32-bit integer at the start to say how many bytes there were per integer. Or if 16 bits were needed, then we could use 2-byte ints, or 3 bytes for 24 bits etc. Worst case is when the documents are bigger than 4MB or so, when we will need 32-bit integers. But that's rare.

So the strategy is pretty simple. Scan all the numbers first to see what is the biggest (or smallest negative number) and work out how many bits we will need. I kind of cheat by rounding this up to the nearest 8 bits, but if you're interested you can refine it to have an arbitrary number of bits. But you won't gain much in compression and you will lose something in speed. Here's my Java code. It just uses ByteBuffer to build arbitrary-sized ints up to 32 bits, in 8-bit hops, and then stores the list as an array of 32-bit ints – compressed, of course. So typically what you'd expect is a 25%-50% reduction in the array size and in some cases 75%. Compressing it further by any significant amount seems to be impossible, given the near-random nature of the data.

I offer no guarantees that this works for all cases etc. But it is freeware. Do what you like with it.

/** * Compress/decompress an array of one to four-byte ints * @author desmond */ public class UnsortedIntCompressor { /** * Find out how many bits we'll need to represent this array of ints * @param maxValue the maximum positive value * @param minValue the minimum negative value or 0 * @return the number of bits needed, rounded to nearest 8, max 32 */ private static int bitsNeeded( int maxValue, int minValue ) { int minBits = 0; int maxBits = 32; if ( minValue == 0 ) minBits = 0; else if ( minValue >= -128 ) minBits = 8; else if ( minValue >= -32768 ) minBits = 16; else if ( minValue >= -8388608 ) minBits = 24; else minBits = 32; if ( maxValue <= 127 ) maxBits = 8; else if ( maxValue < 32767 ) maxBits = 16; else if ( maxValue < 8388607 ) maxBits = 24; else maxBits = 32; return (maxBits>=minBits)?maxBits:minBits; } /** * Compress an array of ints * @param array an array of unsorted ints to compress * @return the compressed array */ public static int[] compress( int[] array ) { int maxValue = 0; int minValue = 0; for ( int i=0;i<array.length;i++ ) { int value = array[i]; if ( value > maxValue ) maxValue = array[i]; else if ( value < minValue ) minValue = value; } int numBits = bitsNeeded( maxValue, minValue ); // the first entry will store the number of bits/8 int numBytes = numBits/8; // plus first entry int len = array.length*numBytes + 4; // 3 is the maximum overrun ByteBuffer buf = ByteBuffer.allocate(len+3); buf.putInt(numBytes); for ( int i=0,j=0;i<array.length;i++ ) { int value = array[i]; if ( numBits>24 ) buf.put((byte)(value>>>24)); if ( numBits > 16 ) buf.put((byte)(value>>>16)); if ( numBits > 8 ) buf.put((byte)(value>>>8)); buf.put((byte)value); } int resLen = (array.length*numBytes)/4 + 1; if ( (array.length*numBytes)%4 != 0 ) resLen++; int[] res = new int[resLen]; res[0] = numBytes; for ( int i=1;i<resLen;i++ ) { res[i]= buf.getInt(i*4); } return res; } /** * Turn a reduced-byte integer array into a 4-byte integer array * @param array the reduced-byte int array previously compressed * @return an array of ints the same size as before it was compressed */ public static int[] decompress( int[] array ) throws NumberFormatException { int numBytes = array[0]; if ( numBytes < 1 || numBytes > 4 ) throw new NumberFormatException("numBytes must be between 1 and 4"); int len = ((array.length-1)*4)/numBytes; int rem = ((array.length-1)*4)%numBytes; if ( rem == 0 ) { int mask = 0xFF<<((numBytes-1)*8); if ( (array[array.length-1]&mask) == 0 ) len--; } ByteBuffer buf = ByteBuffer.allocate((array.length+1)*4); for ( int i=1;i<array.length;i++ ) buf.putInt(array[i]); int[] res = new int[len]; for ( int i=0;i<len;i++ ) { res[i]= buf.getInt(i*numBytes); res[i] = res[i] >> 8*(4-numBytes); } return res; } public static void main( String[] args ) { Random r = new Random(); int[] uncompressed = new int[100]; // adjust the "23" here to 7 or 31 for testing other int sizes int limit = (int)Math.pow(2,23); for ( int i=0;i<100;i++ ) { uncompressed[i] = r.nextInt(limit); } int[] compressed = null; try { compressed = UnsortedIntCompressor.compress(uncompressed); System.out.println("Length of compressed="+compressed.length); int[] decompressed = UnsortedIntCompressor.decompress(compressed); if ( decompressed.length != 100 ) System.out.println("decompression failed. length="+decompressed.length); for ( int i=0;i<100;i++ ) { if ( decompressed[i] != uncompressed[i] ) System.out.println("uncompressed ("+uncompressed[i] +") not equal to decompressed ("+decompressed[i]+")"); } } catch ( NumberFormatException nfe ) { nfe.printStackTrace(System.out); } } }

Tomcat 7 error

A few people seem to be bitten by an error in the Ubuntu distribution of Tomcat7. Basically when you shut down the service it gives the following messages:

INFO: Destroying ProtocolHandler ["http-bio-8080"] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/common/classes], exists: [false], isDirectory: [false], canRead: [false] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/common], exists: [false], isDirectory: [false], canRead: [false] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/server/classes], exists: [false], isDirectory: [false], canRead: [false] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/server], exists: [false], isDirectory: [false], canRead: [false] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/shared/classes], exists: [false], isDirectory: [false], canRead: [false] Dec 18, 2015 1:46:15 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat7/shared], exists: [false], isDirectory: [false], canRead: [false]

Opinions differ on what causes it, but it is pretty clear: Tomcat is assuming that the shared, server and common directories, which are in /var/lib/tomcat7 are in /usr/share/tomcat7. In other words, the configuration has mixed up catalina.home with catalina.base. catalina.base is supposed to be /var/lib/tomcat7 and catalina.home is /usr/share/tomcat7. All you have to do is edit the file /var/lib/tomcat7/conf/catalina.properties so that all references to those directories have the correct prefix and you're good. I ignored the "common.loader" line as changing this created some weird effects. But lower down the file is mixed up. Reboot and you should be good.

Allow local File access via jQuery.ajax in Chrome/Chromium

It is often useful to encapsulate a website onto a local file system without using a web-server. I wanted to create a web-archive of a site and then substitute the jQuery get calls with local file reads. That way I would not need to access the Internet for the web-archive to work, and I could give it to other people on a usb stick, and they wouldn't have to install a webserver to run it. So I thought I'll use jQuery.get or jQuery.ajax to read the local file. They would all be JSON files, since that is what my server returns, but you can tweak it for other formats. After a bit of fiddling I got it right:

<!doctype HTML>
<html><head>
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
<link id="link" rel="external" href="data.json">
<script src="jquery-1.11.3.min.js"></script>
<script>
$(document).ready(function(){
  $.ajax({
    url: "data.json",
    beforeSend: function(xhr){xhr.overrideMimeType("application/json");},
    dataType: "json",
    success: function(data){$("#result").html(data.text)}
  });
});
</script>
</head>
<body>
<p id="result"></p>
</body>
</html>

This reads the local file data.json and sets the contents of the #result element to the value of the "text" property. Here's my data.json just to be clear:

{"text": "Oh my word!"}

You also need a local copy of jQuery. Now this works just dandy on Safari and Firefox and I'm told in IE, but not in Opera or Chromium. Chromium says that this is a cross-origin request. I don't see why. I opened my HTML file using the file:// protocol and that tried to open a file in the same directory using the file:// protocol. How is that cross-origin? Because it made me cross? But the workarounds in Chromium/Chrome are dire: they suggest installing a local webserver – ridiculous. The only reason I am doing this is to avoid that. Or they say use the --allow-file-access-from-files option when launching chromium. But it doesn't work, at least not on Linux. However, I discovered that if you add --allow-file-access as a second option it works, though neither works on its own. I saved the options in /usr/share/applications/chromium.desktop under the Exec= property. It's a bit of a pain to ask people to do that but it is far better than installing a webserver.

Here's my Exec line from chromium.desktop:

Exec=chromium-browser --allow-file-access-from-files --allow-file-access %U

To fix Opera is more sensible: You just set Allow File XMLHttpRequest in UserPrefs of opera:config.

Swapping suffixes on file names in bash

A common problem when writing shell scripts is to swap suffixes for file names. For example, I wanted to translate a batch of markdown files to html but also to apply a sed script so I could get curly quotes and long dashes etc. To do that I needed to create a temporary file, so I had to go from file.md to file.tmp to file.html. Each time I needed to swap suffixes. Having looked around I couldn't find a neat way to do that, and most of them used expr, which starts a new process. I wanted to do it natively in bash or even dash (the default Ubuntu shell). So I wrote a trivial but neat function and a test, which can be stripped out. The function is all you need:

#!/bin/bash
string="banana.md"
function swap {
    echo "${1:0:(${#1}-(${#2}+1))}.$3"
}
swap "banana.md" "md" "tmp"

To use it in a real script just use backticks thus:

...
function swap {
    echo "${1:0:(${#1}-(${#2}+1))}.$3"
}
markdown myfile.md > `swap "myfile.md" "md" "html"`
...