A minor rant on why the advances in computer technology often do not matter much to the end user, because it is still impossible to accomplish simple but unusual tasks. Today's task: inject a 80 MB access.log into a database.
Computer technology has come a long way since I bought my first Amiga almost 10 years ago. Generation after generation, computers became faster, had more RAM and overall promised to do any work just faster.
Fast forward from 1990 to last Sunday. I have an old A3000 standing in a corner of a room, and below my desk there's a P450 running Windows and more and more infrequently, WinUAE. I had been planning to get my server logs from the last three months and inject them into a database, one month at a time.
Approximate size of my March server logs: 80 MB. Maximum RAM in my A3000 with WarpEngine: 64 MB. Maximum RAM in WinUAE emulation: Less than that. So no matter what I do, I have to use the PC, which is equipped with 128 MB.
So while the choice for the hardware has been made (and in fact, I already made that choice for my everyday-hardware platform a year ago), I still had to decide on the software.
The problem: there is no program available that will take an access.log and automatically inject it into a database, so that you can run queries on them. Access just plain rejects anything that is not in comma-separated-values format, and I don't have any other program that could deal with the amounts of data I'd need to handle. A normal spreadsheet will not accept more than about 32000 or 64000 rows, I need half a million.
So the only solution, without writing a custom program, seems to be a text editor with macro capabilities. Should be relatively trivial, shouldn't it? Wrong. 128 MB RAM is apparently not enough to hold both Windows, an 80 MB text file and still have enough room to work in. The solution: Not applying the macro to the global document, but only to a selection at a time. The problem: even running the macro on 50,000 lines at a time takes half an hour each. Saving the file takes five minutes, at least. Since the memory requirements are too high to multitask adequately, it's difficult to get this done. Either I sit in frontg of the computer even though I can't do much, or I go elsewhere and do not notice when it terminates, or runs out of memory.
Even though I had an entire day and an evening, I didn't manage to run the macro ten times without screwing up the format of my server log. Since each attempt adds 30 minutes to the duration of this conversion, undoing the last modifications means losing half an hour of work. I have yet to do any queries in Access, but preliminary test indicate that this, at least, should work.
What's the point?, I hear you scream. My point is that, even though I've now seen ten years worth of promises of better and more advanced computers, that I still can't accomplish the most simple tasks in an adequate period of time.
Computers have indeed come a long way, but it sure doesn't show. While things might be flashier and overall look nicer than 5 or 10 years ago, it is still impossible to use computers for very specific tasks without getting your hands dirty by writing own code.
I'm not sure if there will be any solution to this problem, ever. Will Amiga Inc. provide the framework for more intelligent computers? Will traditional operating systems be expanded to actually cater for specific user needs? Are scripting languages such as Perl, Rebol or VBS the solution for my kind of problem?
I'm not sure.