wiki:dataRepositoryManager

Version 15 (modified by Martin Kolman, 11 years ago) (diff)

--

modRana data repository document

This is a description for a simple program, that should generate a data repository for offline Monav routing data. It should be flexible enough to enable extending it to also handle other data in the future.

Requirements

  • easy repository setup and regeneration

CLI options

Repositories

Repository definition file

This is a JSON file called repository.json that sits at the root of the repository.

Structure

Description of the different sections

= header =
format_version - 1 for now

= repository =
* name - natural language name of the repository
* last_update - epoch of the last update


= data =
Contains sections for the different data sub-repositories.


== monav ==
* name = "Monav offline routing data repository"

=== Example_Package ===
* pack_type = "monav"
* url - pack URL
* last_updated - epoch of the last update
* bytes_size (optional)

==== zsync_file_list - (optional) ===

Monav data repository

Located in the monav driectory in the main repository folder.

The individual packages are stored in a simple folder structure:

continent/[country_name/][city_name/]package_name.tar.gz

NOTE: Square brackets indicate optional path components.

Internal package structure

The package archives contain a named folder and inside this folder is the folder with Monav routin data.

Example - car routing data for Czech Republic:

Czech_Republic/routing_car/

Each package contains data for a single transportation mode. Like this, the users can select to download only the routing data they actually need.

package.JSON

Inside the routing_* subdirectory is JSON file called package.JSON that makes it possible to map the package folders back to packages existing in the repository.

This is mainly needed to facilitate package updates.

= header =
format_version - 1 for now

= package =
origin - base repository URL (ex.: http://data.modrana.org
path - path to the package in the repository (ex.: monav/europe/czech_republic.tar.gz)

Monav data processing

  • the Monav preprocessor can run in multiple threads
    • the repository generator should detect the number of cores and start supply the corresponding number to the preprocessor command line arguments using the -t option

Benchmarks

2012.10.04
france.osm.pbf
bike speed profile PQ only@Asteria:
1 thread = 295.96 s
2 threads = 172.69 s
48 threads = 25.83 s
96 threads = 28.132 s

2012.10.04
czech_republic.osm.pbf
Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
1 thread:
real	0m23.392s
user	0m22.525s
sys	0m0.756s
2 threads:
real	0m18.536s
user	0m26.162s
sys	0m0.700s
8 threads:
real	0m15.879s
user	0m27.570s
sys	0m0.924s

Asteria:
1 thread:

2 threads:

4 threads:

8 threads:

48 threads

96 threads:
real	0m13.937s
user	1m15.157s
sys	0m21.989s