Sqlite batch import script

original TMO post

OK, I've got a working Perl script to import map tiles into the database.

At the moment, it won't run on the N900 itself, as I'm having a few problems getting the SQLite module to build on my scratchbox installation; I'll work on that over the next few days when I have spare time and see if I can get that working.

Dependencies: You'll need to install the Perl DBI and DBD-SQLite modules. These should be available from your disto's repositories (otherwise, get them from CPAN).

It took about 75 minutes to import my 200 000 tiles into the database on my dual-core Athlon machine; obviously this will vary depending on the speed of the machine it's done on. The script is interruptible - just hit Control-C and it'll shut down cleanly. Then restart at your leisure.

Getting it to work on an OS other than Linux is left as an exercise for the reader, as I'm happily Microsoft-free.

It demonstrates well the oft-mentioned clustering problem with FAT partitions; my tile files allocated 12.5GB of space, whereas the store file is only 543MB.


Download the script from (local miror)

Either copy your ~/MyDocs/?.maps directory to your Linux box or mount your MyDocs partition on it. Try to preserve the timestamps if copying (as they're recorded in the database, I assume they're used to work out if a new copy of any tile is needed if they've changed on the tile server) - for example, if using scp:

scp -pr ~/MyDocs/.maps mycomputer:/tmp

The script only works on tiles in the current working directory, so for each map-type, cd into the relevant directory:

cd /path/to/tile_directory

Then run the script (no parameters needed).

If necessary, copy the lookup.sqlite and store.sqlite.* files back to your N900.

If there are any tiles already in the database, it leaves them there and it doesn't try to over-write any if one it finds already exists (at some time I should change this so it does over-write if the tile file is newer).

If you already have tiles in a database, I recommend making a copy first, just in case anything goes wrong. I've tested the script a fair bit, but can't guarantee something won't screw up.

Last modified 12 years ago Last modified on May 20, 2012, 12:16:34 PM

Attachments (1)

Download all attachments as: .zip