Jekyll2016-08-22T22:10:54+00:00http://droftware.github.io/My adventures/mis-adventures with softwareThe How part (6) - Concatenating OSM way chunks2016-08-22T23:30:00+00:002016-08-22T23:30:00+00:00http://droftware.github.io/way-concatenation<p>( This post is related to my GSoC project with Marble. For more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<h4 id="problem">Problem</h4>
<p>As of now, If you load an openstreetmap file containing linestrings such as highways, lanes or streets, in Marble and zoom to level 11, you will find that the highways or streets are not contiguous and appear as broken.</p>
<p>You will find the roads/streets as broken</p>
<p><img src="http://droftware.github.io/assets/ak1.png" alt="roads_broken" /></p>
<p>Instead of getting contiguous roads</p>
<p><img src="http://droftware.github.io/assets/ak2.png" alt="road_joined" /></p>
<h4 id="cause">Cause</h4>
<p>One of the primary reasons for this discontiguity is that often a particular highway/lane/street is described using multiple contiguous OSM way elements instead of a single OSM way element. Marble treats each of these specific way element as a linestring object. However Marble omits rendering any objects which are smaller than 2px, in order to improve rendering performance. Due to this, many of the OSM way elements, which are smaller than 2px don’t get rendered. Hence the street appears broken since only those of its OSM way elements are rendered which are larger than 2px.</p>
<p>One of the reasons which I can think of and which justifies this highway description using multiple elements is that a street might be called by a certain name for a specific stretch and might be called by some other different name for the remaining stretch. However, at level 11 we are mostly concerned with the type of highways (primary, secondary, motorway , residential) rather than the specifics such as names.</p>
<p>Usually, the multiple OSM way elements of a single highway share the first and the last node ID references. For example consider <1n…2n> as an OSM way element where 1n and 2n corresponds to the first and last node IDs of the way. A highway described by an ordered list of node IDs 1 to 56 can then usually be represented by the following OSM way elements <1…5>, <5…13>, <13…28>, <28…36>, <36…56></p>
<p>I exploited this way of representation to create a way concatenating module in the existing osm-module tool. For the above example, the module would concat all the 5 OSM way elements into a single OSM way element <1…56></p>
<div class="highlighter-rouge"><pre class="highlight"><code>osm-simplify -t highway=* -w input.osm
</code></pre>
</div>
<p>The above command concatenates all the highways present in the input file and produces a new osm file as output.</p>
<p>Apart from solving the problem of discontinuity, way concatenation also results in data reduction since it is eliminating redundant node references and way elements. This data reduction in turn results in faster file parsing as well as improved rendering performance since now to render a single stretch of highway one only needs to create and render a single linestring object as opposed to multiple linestring objects.</p>
<h4 id="algorithm">Algorithm</h4>
<p>The tricky part of coding this OSM way concatenator is actually coming up with an algorithm which concatenates all the linestrings present in an input file. Finally I and my mentors Dennis Nienhüser and Torsten Rahn were able to come up with a working algorithm for concatenating osm ways of a file in reasonable time (approximately O(n) time).</p>
<p>The algorithm involves a made up data structure called WayChunk which basically is a list of contiguous ways. It also stores a GeoDataVisualCategory which represents the type of linestring. For example in case of highways GeoDataVisualCategory will contain the kind of highway, whether it is a motorway, primary, secondary or a residential type of highway.</p>
<p>The algorithm utilizes a multi-hash-map to bundle together OSM way elements which share a common starting or terminating node. This multi hash map has nodeID’s as keys and WayChunk’s as the values. The idea is that at any given instant, this map will contain the starting and ending point of the ways which have been evaluated till now, as keys and the corresponding way chunk as the value pointed to by these keys. Now whenever we encounter a new way, and if it’s starting or ending node matches with any of the existing way chunks as well as the type i.e. GeoDataVisualCategory of the way matches with the type of the way chunk, then this way is added to the way chunk and the values of the map are adjusted so that the map’s keys are the starting or ending nodes of some way chunk and not the intermediary ones. This way, eventually, we are able to concat the multiple small OSM chunks of highways, railroads into singular way elements.</p>
<p>The reason we are using multi-hash-maps instead of regular hash maps is that at a particular node, two or more highways(linestrings) of different types may emanate or terminate. Hence a given node may be associated with two or more way chunks having different type(GeoDataVisualCategory).</p>
<p>The algorithm in more detail is described below:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Concat(input_osm_file):
Iterate over all of the way elements having key=value tags specified during input
Check if the first or the last node ID of the way is present in the multi-hash-map.
If neither of the IDs are present
CreateWayChunk(way)
If only the first ID is present in the map
Check if any chunk exsits in the map which has the key as that of first ID and GeoDataVisualCategory as that of the way.
If such a chunk exists
Append the way to this chunk accordingly(reverse the way if required)
Delete the first ID from the map
Insert a new entry into the multi map having the key as last node ID of the way and value as that of the found chunk
If not
CreateWayChunk(way)
If only the last ID is present in the map
Check if any chunk exsits in the map which has the key as that of last ID and GeoDataVisualCategory as that of the way.
If such a chunk exists
Append the way to this chunk accordingly(reverse the way if required)
Delete the last ID from the map
Insert a new entry into the multi map having the key as first node ID of the way and value as that of the found chunk
If not
CreateWayChunk(way)
If both the IDs are present in the map
Check if any chunk exsits in the map which has the key as that of first ID and GeoDataVisualCategory as that of the way.
Check if any chunk exsits in the map which has the key as that of last ID and GeoDataVisualCategory as that of the way.
If both the chunks exist
Append the way and the second chunk to the first chunk and modify the map accordingly
If only first chunk exists
Append the way to the first chunk and modify the map accordingly
If only last chunk exists
Append the way to this last chunk and modify the map accordingly
If none of the chunks exist
CreateWayChunk(way)
Finally iterate over all the WayChunks and merge the ways in each of their lists to form one single osm way for each chunk
CreateWayChunk(way):
Create a new WayChunk
Append the way to this new way chunk
Set the GeoDataVisualCategory of the chunk as that of the way
Insert two new entries in the multi map having the starting and ending node IDs of the way as keys and the created WayChunk as the value.
</code></pre>
</div>
<h4 id="results">Results</h4>
<p>The first image having discontiguous highways represents the raw input OSM file. This file has 2704 ways in total and has a size of 4.7 MB</p>
<p>The way-concatenator reduced the number of ways to 812 and the size to 2.9 MB. The second image having contiguous roads represent the output of the way concatenator.</p>
<p>If we remove the redundant nodes from the above output using the node reducing module described in the previous post, we get a resulting file having a size of 2.5MB. This node reducer removes 15146 redundant nodes (keeping the resolution at level 11).</p>
<p>If you suspect that due to node reduction there will loss in quality of rendering, then look at the below rendered image and compare it with the above ones.</p>
<p><img src="http://droftware.github.io/assets/ak3.png" alt="node_road_joined" /></p>
<p>The node reducer and the way concatenator have resulted in a size reduction of approx 46% for the above sample without any significant loss in rendering quality.</p>( This post is related to my GSoC project with Marble. For more information you can read the introductory post )The How part (5) - Removing redundant nodes from OSM files2016-08-21T21:00:00+00:002016-08-21T21:00:00+00:00http://droftware.github.io/removing-redundant-nodes<p>( This post is related to my GSoC project with Marble. For more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<p>The requirement for medium level tiles are a little bit different from those of lower level tiles. The biggest difference being that the medium levels will use openstreetmap data as opposed to the Natural Earth data. Another difference is that in lower level tiles the major data pre-processing steps were concatenation and conversion since the data was in shapefile format and the requirement was that of OSM whereas in medium levels the major data-pre processing steps are reduction, simplification, filtration and merging.</p>
<p>As already mentioned in a previous post, openstreetmap describes data at a pretty high resolution. Such resolution is fine for higher levels such as street levels but cannot be suitably used for medium (and also lower levels). Hence data-preprocessing steps such as reduction to a lower resolution, filtering features such as trees and lamp posts which are not visible at medium levels , and merging of buildings and lanes which are very close to each other.</p>
<p>In order to pre-process data so as to reduce, filter, combine; various tools need to be built which modify the raw openstreetmap data so as to make it suitable for a particular Marble zoom level.</p>
<p>The first tool which I built removes redundant nodes from geographical features present in the input OSM file. Redundant nodes are the nodes which are just not required at a particular zoom level since the resolution with which these nodes describe their parent geographic feature exceeds the maximum resolution which is visible at a particular zoom level.</p>
<p>Consider this particular patch of lanes at zoom level 11.</p>
<p><img src="http://droftware.github.io/assets/kpiece1.png" alt="kpiece1" /></p>
<p>Now observe the number of nodes with which this patch has been described.</p>
<p><img src="http://droftware.github.io/assets/kpiece2.png" alt="kpiece1" /></p>
<p>A small square describes a single node. These squares are colored in a translucent manner so as to depict overlappings. Higher the intensity of the color, greater are the number of overlappings.</p>
<p>As you can clearly see, these many nodes are not required for proper rendering of OSM data at medium zoom levels. In order to overcome this problem, I wrote a node reducing module. This module is a part of osm-simplify which is an OSM data preprocessor jointly being developed by Dávid Kolozsvári and me which supports many different kinds of OSM data preprocessing such as clipping, tiling, way-merging(other simplification tools are still being developed)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>osm-simplify -z 11 -n input.osm
</code></pre>
</div>
<p>The above command removes the redundant nodes according to the resolution of level 11 of Marble and produces a new reduced OSM file as output.</p>
<p>The underlying algorithm of this node reduction module is pretty simple</p>
<ul>
<li>Create a new empty list of nodes called reducedLine</li>
<li>nodeA = first node of the linestring, ring, or a polygon which is under consideration.</li>
<li>add nodeA to reducedLine</li>
<li>Iterate from the second to the second last node (so as to retain the first and last nodes) of a linestring, ring, or a polygon
<ul>
<li>If the great circle distance between the nodeA and the currentNode is greater than the minimum resolution defined for that level</li>
<li>Then add currentNode to reducedLine and change value of nodeA to that of currentNode</li>
</ul>
</li>
<li>Add the last node to reducedLine</li>
<li>return reducedLine</li>
</ul>
<p>This simple tool results in significant removal of redundant nodes. The final goal is that this node reduction module along with other modules of osm-simplify tool will result in a significant size reduction of the OSM data for a particular zoom level resulting in improved rendering performance of Marble, without compromising much on the visual quality of the rendered data.</p>
<p>In the next post, I will describe about way-merging module as well as do a little bit of objective comparison of the reduction in size of OSM data caused by these modules.</p>( This post is related to my GSoC project with Marble. For more information you can read the introductory post )The How part (4) - Automating tile generation for lower levels2016-08-21T18:00:00+00:002016-08-21T18:00:00+00:00http://droftware.github.io/tiles-generator<p>( This post is related to my GSoC project with Marble. For more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<p>After revamping shp2osm tool and adding the missing styles, the only thing remaining for supporting lower level tiles was automating the process of tile generation . Before the process was automated, in order to create the lower level tiles one needed to manually download the Natural Earth shapefiles, combine and convert the shapefiles into a single OSM file using shp2osm tool, and then using the existing vector tile creator of Marble (which divides a large OSM file into small chunks of OSM vector tiles) create the tiles.</p>
<p>I wrote a script which simplifies and automates the process of generating lower level tiles for Marble. The script takes a text file as input. This text file includes a listing of the zoom levels and the Natural Earth geographical features which are to be included in those zoom levels.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>*0
ne_110m_land
ne_110m_geographic_lines
*1
ne_110m_land
ne_110m_geographic_lines
ne_110m_admin_0_boundary_lines_land
*3
ne_110m_land
ne_110m_geographic_lines
ne_110m_admin_0_boundary_lines_land
ne_110m_admin_1_states_provinces_lines
ne_110m_admin_0_pacific_groupings
ne_110m_rivers_lake_centerlines
ne_110m_lakes
ne_10m_bathymetry_K_200
ne_10m_bathymetry_G_4000
</code></pre>
</div>
<p>As you can see, the above input file describes what all Natural Earth features are to be included for 0th, 1st and 3rd levels of the vector OSM theme of Marble</p>
<p>Using the above input file, the script checks the input directory for the Natural Earth geographical features specified in the file. If any geographical feature is not present, the script downloads the files in zipped form and then unzips it. Once all the required Natural Earth features are present in the input directory, using the shp2osm tool it combines all the data for a particular level and creates a single OSM file containing all the Natural Earth features specified in the input file. Then using the existing vector tile creator, it creates all the required OSM tiles , arranged properly in respective folder and sub-folders for the levels specified in the input file.</p>
<p>With this tool in place, we can now easily generate the lower level vector tiles for Marble.</p>
<p><img src="http://droftware.github.io/assets/lev31.png" alt="level_3" />
<em>level 3</em></p>
<p><img src="http://droftware.github.io/assets/lev51.png" alt="level_5" />
<em>level 5</em></p>
<p><img src="http://droftware.github.io/assets/lev71.png" alt="level_7" />
<em>level 7</em></p>
<p><img src="http://droftware.github.io/assets/lev91.png" alt="level_9" />
<em>level 9</em></p>
<iframe width="854" height="480" src="https://www.youtube.com/embed/4pEJYOS62KQ" frameborder="0" allowfullscreen=""></iframe>
<p><a href="https://youtu.be/4pEJYOS62KQ">video</a></p>
<p>As you can see in the video, as all the data is getting rendered dynamically we are getting a performance lag and the navigation is not very smooth. This is because of improper tile cutting as well as the huge size of data getting rendered at a particular instant. For creating the tiles, the script itself the preexisting vector tile cutter of Marble. This vector tile cutter in turn depends on osmconvert to actually clip the data and create smaller OSM tiles. However osmconvert’s clipping is not very exact resulting in data redundancy. This redundancy causes Marble to render the same geographical feature 3 or 4 times causing severe performance penalties. Currently the clipping is being improved by Dávid Kolozsvári who is also a GSoC student working on Marble.
Apart from the clipping, there are resolution issues i.e. a particular feature is described by much greater number of nodes than required.</p>
<p>After all the work related to lower level tiles was done, began the second part of my project which dealt with creation of tools for simplification and reduction of OSM data so that it is rendered smoothly and according to the visual requirement of medium zoom levels. I will discuss about these tools in future posts.</p>( This post is related to my GSoC project with Marble. For more information you can read the introductory post )The How part (3) - Concatenating the Natural Earth geographical features into a single OSM file2016-08-21T16:00:00+00:002016-08-21T16:00:00+00:00http://droftware.github.io/concat-features<p>( This post is related to my GSoC project with Marble. For more more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<p>Till now, the polyshp2osm tool, which I had modified to support various kinds of geometries as well as OSM tag-mappings for Natural Earth metadata, supported conversion of a single shapefile into OSM.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>python3 ~/a/marble/tools/shp2osm/polyshp2osm.py ne_110m_rivers_lake_centerlines.shp
</code></pre>
</div>
<p>The above command will convert the given shp file containing rivers and lakes into its OSM equivalent.</p>
<p>One of the major aims of my project was to create a tool for automatically generating the OSM tiles for lower zoom levels using Natural Earth data. In my previous post, I described how I added styling for many geographical features so as to enable rendering of many Natural Earth features such as reefs and bathymetries. Now before creating a tool which can output the tiles, I need a tool which can concatenate all the different Natural Earth features such as roads, rails, rivers, reefs into one single OSM file which then can be broken down into tiles for lower zoom levels. Basically I need to modify the tool so that it can take multiple shp files as input and then produces a combined OSM file.</p>
<p>Now the straightforward way of doing this is to just concat the XML elements produced for different shp files. This will obviously work but will contain way too much redundant data especially in the form of redundant nodes. I solved this redundancy by using a dictionary which maps the (longitude, latitude) of a node to a unique node ID. Now whenever the tool iterates over the nodes of a polygon or a linestring in the tool, it is checked whether that node is already present in the dictionary or not. If it is not present, we assign a new ID to the node and add it to the dictionary, if it is present in the dictionary , we use the node ID of the existing node instead of creating a redundant node having the same longitude,latitude but a different ID.</p>
<p>Now we can give multiple shp files as input to the tool and it will generate a single OSM file containing all the features of the input shp file without any redundant node data.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>python3 ~/a/marble/tools/shp2osm/polyshp2osm.py ne_50m_land ne_50m_geographic_lines ne_50m_lakes ne_50m_glaciated_areas
</code></pre>
</div>
<p>The above command will concatenate the Natural Earth shp files containing land, geographic line, lakes and glaciated areas into a single OSM file.</p>( This post is related to my GSoC project with Marble. For more more information you can read the introductory post )The How part (2) - Adding styling for various geographical features2016-08-21T12:00:00+00:002016-08-21T12:00:00+00:00http://droftware.github.io/how-styles<p>( This post is related to my GSoC project with Marble. For more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<p>As I mentioned in my earlier post, I modified the polyshp2osm tool so as to support conversion of Natural Earth shapefiles to OSM format so that Marble is able to render geographical features(roads, rivers, reefs, airports) provided by the Natural Earth data set. Now the primary problem was that Marble never supported the rendering of many of the geographical features provided by Natural Earth. In order to enable rendering of a new geographic feature(reef, boundaries), one needs to describe how the feature must be styled by describing the colors to be used, the width of the boundaries, any geometric patterns or any icon which needs to be associated with the geometry.</p>
<p>Before describing my work let me first describe how Marble handles the styling for the vector OSM theme. Marble reads the OSM file and for any geographical feature (roads, rivers, reefs, airports, shops, buildings etc) assigns it a GeoDataVisualCategory value. The geometry associated (point, polygon, linestring ) with the geographical feature is primarily styled according to this GeoDataVisualCategory. Depending on the presence of other OSM key=value pair tags and the current zoom level of the map, Marble might dynamically change the styling.</p>
<p>In fact, this ability to dynamically change the styling was recently added by my mentor Dennis Nienhüser during the Randa meetings.</p>
<p>In most of the cases, for adding the styling for a new geographic feature in Marble, one needs to do the following steps</p>
<ul>
<li>Create a new GeoDataVisualCategory</li>
<li>Assign a default style to this GeoDataVisualCategory by describing the colors, width, pattern, icon which are associated with the feature</li>
<li>Decide the order of this new visual category w.r.t other visual categories. Visual categories which have a higher priority will be drawn on top of visual categories which have a lower priority.</li>
<li>Decide the minimum zoom level at which this new visual category starts appearing.</li>
<li>Map this visual category to some corresponding OSM key-value pair.</li>
<li>Decide if you want to change the default styling associated with this GeoDataVisualCategory on some particular zoom level or when some particular OSM key=value tags are encountered.</li>
</ul>
<h2 id="natural-reefs">Natural Reefs</h2>
<p>Has an OSM equivalent natural=reef which I used for tag mapping it to the styling of reefs.</p>
<p><img src="http://droftware.github.io/assets/reef.png" alt="reefs" /></p>
<h2 id="landmass">Landmass</h2>
<p>Natural Earth provides the data for countries, continents, urban areas, islands as polygons. In order to provide a common tag mapping for
these land polygons, I constructed a custom marble_land=landmass for tag mapping the above land polygons to the styling of rendered land areas in Marble.</p>
<p><img src="http://droftware.github.io/assets/landmass.png" alt="landmass" /></p>
<h2 id="administrative-boundaries">Administrative Boundaries</h2>
<p>The OSM administrative boundaries have an admin_level=x tag which defines the size/importance of a particular administrative boundary. I tag mapped these admin_level=(1..11) tags to 11 different GeoDataVisualCategories .</p>
<p><img src="http://droftware.github.io/assets/admin.png" alt="admin_boundaries" /></p>
<h2 id="salt-lakes">Salt Lakes</h2>
<p>Instead of creating a new visual category, I added an additional OSM tag salt=yes to the list of key=value tags list of salt lakes while keeping the visual category as NaturalWater. Now when a salt lake in encountered during parsing, Marble assigns it the default styling of NaturalWater which is a shade of blue with blue outline. However due to the tag salt=yes the styling changes to a shade of yellow with a dashed blue line.</p>
<p><img src="http://droftware.github.io/assets/sl.png" alt="salt_lakes" /></p>
<h2 id="antarctic-ice-shelves">Antarctic Ice Shelves</h2>
<p>I created a new visual category NaturalIceShelf for styling the antarctic ice shelves and tag mapped it to natural=glacier and glacier:type=shelf</p>
<p><img src="http://droftware.github.io/assets/shelf9.png" alt="ice_shelves" /></p>
<p><img src="http://droftware.github.io/assets/glac9.png" alt="glacier_ice_shelves" /></p>
<h2 id="marine-indicators">Marine Indicators</h2>
<p>I tag mapped the OSM tag maritime=yes to a new visual category BoundaryMaritime whereas used the tags (boundary=administrative, admin_level=2, maritime=yes, border_type=territorial) for dynamically changing the styling of disputed marine boundaries.</p>
<p><img src="http://droftware.github.io/assets/marine_indic.png" alt="marine_indicators" /></p>
<h2 id="international-date-line">International Date Line</h2>
<p>Created a custom OSM tag marble_line=date and tag mapped it to a new visual category InternationalDateLine.</p>
<p><img src="http://droftware.github.io/assets/idl.png" alt="idl" /></p>
<h2 id="bathymetries">Bathymetries</h2>
<p>Styling Bathymetries was quite tricky since I was allowed to use only a single visual category for styling different kinds of bathymetries. Initially I thought that this was going to be straightforward and I only need to pass the elevation info of the bathymetry and depending on this info, I can then dynamically change the styling. However I eventually found that in order to do the above task, I need go down a layer of abstraction and hack a few things at the graphics level.
The problem was that the bathymetries at level 200 and 4000 were getting mixed up. Ideally bathymetries at level 4000 should appear above the bathymetry at level 200 but since the z-value(which decides which feature must be rendered on top of others) is fixed for a particular GeoDataVisualCategory, rendering the bathymetries was getting messed up. I had to make special cases in the graphics related methods of Marble so as to circumvent this problem.</p>
<p><img src="http://droftware.github.io/assets/bathymetry1.png" alt="bathymetry1" /></p>
<p><img src="http://droftware.github.io/assets/bathymetry2.png" alt="bathymetry2" /></p>( This post is related to my GSoC project with Marble. For more information you can read the introductory post )The How part (1) - Converting Natural Earth SHP to OSM format2016-08-03T21:00:00+00:002016-08-03T21:00:00+00:00http://droftware.github.io/how-shp-to-osm<p>( This post is related to my GSoC project with Marble. For more information you can read the <a href="/why-part-gsoc/">introductory post</a> )</p>
<p>So my first GSoC task was finding efficient mechanisms/procedures for converting Natural Earth data in SHP format to OSM format. My mentors gave me two options, either create a new tool from scratch using Marble libraries on the lines of the existing shp2pn2 converter or modify the existing polyshp2osm tool. On IRC, it was discussed that if polyshp2osm.py script does not offer any advantage in extracting the metadata /semantics from SHP to OSM format resulting in suitable rendering of lakes and roads, then a new tool must be built.</p>
<h2 id="prep-work">Prep Work</h2>
<p>After experimenting with the polyshp2osm tool, I figured out the following things about it.</p>
<p>polyshp2osm is built upon the GDAL/OGR python bindings. GDAL/OGR is a library for reading/writing raster and vector geospatial formats. It presents a single abstract data model, for all the formats, to the calling application. polyshp2osm reads the SHP file into this abstract data model and then loops over all the possible geometries to figure out the nodes, ways and relations of the corresponding OSM file. In this regard it is quite similar to the shp2pn2 tool which reads the SHP file into a GeoDataDocument.
However, polyshp2osm is limited in the sense that it provides support only for polygons and not for linestrings and other geometries. In fact, it throws an error if we try to convert SHP files containing linestrings like ne_10m_roads and ne_10m_rivers_lake_centerlines. Thus polyshp2osm lags in geometry conversion and effort must be put in order to perform this conversion properly.</p>
<p>The other area of polyshp2osm is its metadata extraction and tag-mapping. The GDAL/OGR library, on which the converter is built provides direct methods for extracting the metadata for a particular geometry present in SHP file. This metadata is available in the form of key-value pairs. These key-value pairs obtained from the SHP files can be directly mapped to corresponding OSM key-value pairs. However all such mapping from the Natural Earth metadata to the OSM key-value pair has to be put into the polyshp2osm program manually since the polyshp2osm was built keeping in mind the MassGIS OpenSpace dataset and has mappings corresponding to this dataset only. Though the tool(polyshp2osm) provides a framework for tag-mapping, still effort must be put in analyzing the metadata provided by different datasets of Natural Earth and finding the corresponding OSM key-value mappings.</p>
<p>polyshp2osm is basically a template code which needs to be extended so that all types of geometries can be converted to OSM. Also, in-spite of having a mechanism for tag-mapping from SHP metadata to OSM key-value pair, it has no inbuilt mapping for NaturalEarth SHP data to OSM key-value and hence this mapping has to be added in.</p>
<p>However, eventually I settled on tweaking polyshp2osm instead of building a new converter from scratch on the lines of shp2pn2. Building a new converter will involve adding support for creating nodes, ways and relations on the existing shp2pn2 procedures which will take more time as compared to adding support for 1-2 more geometries in polyshp2osm. Also I will have to build a metadata extraction and tag-mapping feature for the new converter which are already present in the polyshp2osm tool.</p>
<h4 id="tweaking-and-fixing-polyshp2osm-so-as-to-make-it-compatible-with-natural-earth-data-and-marble-stylings">Tweaking and fixing polyshp2osm so as to make it compatible with Natural Earth data and Marble stylings</h4>
<p>So basically there were three types of additions/modifications which I had to do</p>
<ul>
<li>
<h3 id="adding-support-for-all-the-geometries">Adding support for all the geometries</h3>
</li>
</ul>
<p>As mentioned before, polyshp2osm supported conversion of only those geographic features which have a polygon type of geometry. However Natural Earth data consisted of all forms of geometries like linestrings, multilinestrings, multipolygons and points (all these geometries are defined by the SHP format). Apart from this, the entire code was written in a script like manner and was not very modular. So before adding any support for remaining geometries I refactored the code to make it a little modular(although as of now it is still not pythonic and is not OOP based) so that code components related to extracting metadata; mapping to OSM key-value pairs; writing OSM node, way , relations elements can be reused for different geometries. After this refactoring, the task of adding additional SHP geometries became straightforward.</p>
<ul>
<li>
<h3 id="mapping-metadata-to-osm-key-value-pairs">Mapping metadata to OSM key-value pairs</h3>
</li>
</ul>
<p>This task involved mapping Natural Earth metadata to the corresponding OSM key-value pairs so that Marble is able to render all the features in the resulting OSM file appropriately. I had to write a dictionary, consisting of around 120 entries which contained all the corresponding metadata to key-value mappings. Using this dictionary the program inserts appropriate key-value tags in the resulting node, way or relation.</p>
<ul>
<li>
<h3 id="fixes">Fixes</h3>
</li>
</ul>
<p>Problems started to emerge when the converted files were used to make vector tiles for Marble. In the initial version of the tiles few of the geographical features such as land masses were not visible. It was eventually found that the converter is not outputting in the correct format. According to http://wiki.openstreetmap.org/wiki/OSM_XML, in an OSM file there should be a block of nodes then a block of ways finally followed by a block of relaitions. The tool was outputting nodes, ways and relations in a random order.</p>
<p>After fixing this, everything was getting rendered suitably however osmconvert, which is used to cut the OSM file into tiles continued to show warnings, sequence errors. This was because osmconvert expects the elements in the OSM file to be in non-decreasing ID’s.</p>
<p>With these two fixes in place, things have been fine till now.</p>
<p>So this was all about shp to osm conversion. In the next post I will try to explain about the various style related changes and the additional geographical features which I have added in Marble.</p>( This post is related to my GSoC project with Marble. For more information you can read the introductory post )The Why part - medium and low tile levels in Marble2016-08-03T17:00:00+00:002016-08-03T17:00:00+00:00http://droftware.github.io/why-part-gsoc<p>It has been quite some time since I have written anything about my GSoC project. Anyways, here it is.</p>
<p>So what exactly is my project and what I am supposed to do?
Let’s start with my project title</p>
<p><em>“Support for medium and low tile levels in the OSM Vector Map of Marble”</em></p>
<h3 id="background">Background</h3>
<p>To the uninitiated it sounds quite cryptic so let me break it down and explain it a little better.
Marble, which is our awesome virtual globe, can display both raster as well as vector maps. Raster maps are made up several images(JPG or some other raster format) arranged in a grid like manner. Vector maps on the other hand are made up of chunks of vector data(XML, JSON etc)., each chunk having geographic information for a particular bounded area.</p>
<p>Depending on the zoom level, Marble loads in geographic data(raster or vector). As we zoom in further and further, the resolution of the geographic information being displayed/rendered increases. Now Marble just can’t load the entire Earth’s data for a particular zoom level since beyond a certain resolution this data set goes into the range of tens and then hundreds of gigabytes. Marble resolves this problem via a geo-data storage/indexing strategy known as QuadTiles. In this method, depending on the zoom level, we divide the entire world into tiles. As the zoom level increases, the number of tiles as well as the resolution of these tiles increases. The advantage is that , when viewing a specific region of earth at a particular zoom level, only a few tiles are required as opposed to loading the entire dataset into memory.
OSM Vector Map is a a new Marble map theme which is still being worked upon, and the cool thing about this theme is that it utilizes openstreetsmap vector data to actively render and display the maps resulting in more crisp, dynamically styled maps.</p>
<h3 id="my-part">My part</h3>
<p>Coming back to my project, I , along with my mentors Dennis Nienhüser and Torsten Rahn and fellow summer of code student David, am supposed to add support for lower and medium zoom levels to this OSM Vector map theme. Before the project started, the map supported high zoom levels and the expectation is that by the end of the project, Vector map theme will fully support lower and medium zoom levels.</p>
<p>Support is a pretty vague term so let me be a little more specific. Till now Marble was able to render/display only the higher zoom levels of OSM Vector map theme. For being able to properly render the lower and medium zoom levels quite a many things need to be done which one does not need with the higher zoom levels.
A question which might be bothering and which even bothered me was that if Marble was able to render higher zoom levels then why don’t we render medium and lower levels on similar principles ?</p>
<p>It is because of the level of detail with which everything is described in the openstreetmap data. For example osm data captures all kinds of roads, highways, bilanes, their nooks and corners; very detailed shapes for buildings. Right now, the higher levels i.e the street levels of VectorOSM directly use OSM data, which is filtered to exclude many elements. We cannot directly use OSM data for lower and medium levels and it is because of the following 2 reason.</p>
<ul>
<li>
<h3 id="visual">Visual</h3>
</li>
</ul>
<p>For the lower and medium zoom levels, which you can compare to a bird’s eye view, we don’t need such detailed and precise data. At that level things are quite different. One does not observe some smaller streets and bi-lanes; multiple lane highways appear as a singular strip; nearby railway lines appeared merged, blocks of buildings appeared joined together, many low level details like poles, trees etc are just not visible. Also, since OSM data is street level data, if directly loaded it will lead to severe screen clutter. Hence we need to pre-process data so as to concatenate, cluster and merge the data elements to make them visually apt for a particular level.</p>
<p><img src="http://droftware.github.io/assets/ak1.png" alt="roads" /></p>
<blockquote>
<p>When OSM data directly loaded and observed from level 11;
This happens because a single highway can consist of several tiny OSM Way elements. Marble omits OSM elements
which are less then 2px on the screen for performance reasons.</p>
</blockquote>
<p><img src="http://droftware.github.io/assets/buildings.png" alt="buildings" /></p>
<blockquote>
<p>Buildings can be clustered and then merged so as to form continous blocks</p>
</blockquote>
<ul>
<li>
<h3 id="performance">Performance</h3>
</li>
</ul>
<p>As told earlier, the VectorOSM map theme directly renders the data and does not depend on pre-rendered content. Now if we directly feed in the OSM data there will be several performance issues since depending on the region being viewed, it may involve rendering of thousands or even millions of OSM elements. Hence we need to reduce the redundancy in OSM data by eliminating nodes as well as concatenating ways wherever possible.</p>
<p><img src="http://droftware.github.io/assets/karlsruhe_clear.png" alt="clear" /></p>
<p><img src="http://droftware.github.io/assets/karlsruhe_cluttered.png" alt="cluttered" /></p>
<blockquote>
<p>Each OSM node is a small square. See the square overlappings as redundancies which need to be removed</p>
</blockquote>
<p>For doing this we are constructing an OSM data pre-processor which concatenates, eliminates, merges, does clustering so as to make OSM data suitable for live rendering for some particular level of Marble.</p>
<p>Also, it was decided that for the lower levels we will use Natural Earth data instead of the regular openstreetdata. It is because Natural Earth contains data which is already filtered and categorized and is available in three resolutions 110m, 50m and 10m which we can easily adapt and style to the needs of Marble. However doing this requires two major things</p>
<ul>
<li>
<h3 id="conversion-from-shapefile-to-osm-format">Conversion from Shapefile to OSM format</h3>
</li>
</ul>
<p>Natural Earth vector data is present in SHP(Shapefile) format. However our Vector OSM theme requires data in OSM format. Hence we need to convert the data from SHP to OSM. There are existing tools which do this conversion but these tools lack in geometry support as well as the exact mapping of Natural Earth to OSM data. These tools need to be extended so that they are able to properly interpret Natural Earth data as well as map them to the OSM format.</p>
<ul>
<li>
<h3 id="styling">Styling</h3>
</li>
</ul>
<p>Stylings for various Natural Earth geographical features need to be added to Marble, some of which have an OSM equivalent like administrative boundaries and some which don’t have any OSM equivalent like Bathymetries. For doing this some Marble specific OSM tags have to be introduced. Also, the stylings must be in sync with the existing OSM stylings.</p>
<p><img src="http://droftware.github.io/assets/earthne.png" alt="globe" /></p>
<blockquote>
<p>Administrative boundaries, bathymetries, IDL , land polygon stylings were added</p>
</blockquote>
<p>In this post, I have mostly described the <strong>Why</strong> part of my project. Over the next few posts I will try to write about the <strong>How</strong> aspect i.e. how I achieved a particular goal and why did I do it using that particular method.</p>It has been quite some time since I have written anything about my GSoC project. Anyways, here it is.Google Summer of Code with Marble !2016-05-20T10:18:00+00:002016-05-20T10:18:00+00:00http://droftware.github.io/marble-gsoc<p>Hello everybody,
I am Akshat Tandon and I am a computer science undergraduate at
IIIT-H, India.
I have been selected in GSoC and this summer I will be working on Marble to improve its vector tiling and rendering capabilities.</p>
<p><img src="http://droftware.github.io/assets/marble-logo-2.png" alt="gsoc logo" /></p>
<p><em>Marble, part of KDE software suite, is an open source virtual globe with tons of awesome features.</em></p>
<h3 id="current-state-of-things">Current state of things</h3>
<p>As of now Marble supports rendering of OpenStreetMap vector data for
higher tile levels. In order to make the vector OSM map theme complete and
end user ready, support for medium and lower tile levels must be added.</p>
<h3 id="what-needs-to-be-done">What needs to be done</h3>
<p>For supporting low level rendering( levels 1, 3, 5, 7, 9), Marble should
render using the Natural Earth vector dataset instead of the OSM vector
dataset. The reason for this is that Natural Earth dataset has categorically
arranged and filtered data which is required for step by step rendering of
lower levels. Apart from this the rendered data should be styled so as to
match the OSM theme and tiling support must be added on the server and
client side.</p>
<p>For the medium level tiles, rendering must be performed using the OSM
dataset. OSM dataset gives us many small chunks and nodes for a
particular geographic data item. These chunks and nodes need to be
filtered and reduced to a manageable size for each specific level so as to
get optimal performance and less screen clutter.</p>
<p>Rendering of higher level tiles need to be improved by adding support for
rooftypes and texture tags of buildings as well as by adding support for
street markers.</p>
<h3 id="my-part-of-work">My part of work</h3>
<p>My work will majorly involve adding vector tile support at low levels using the Natural Earth dataset.
This dataset is provided in SHP format and hence prior to any coding, the vector data in SHP format will need to be
converted into OSM format using pre-existing tools such as shp2osm. After this we will have the OSM files but they are not yet tiled. These files would contain an enormous amount of data and if we load all this data into Marble then it would go out of memory and any color or styling would not exist.</p>
<p><img src="http://droftware.github.io/assets/marble_sc1.png" alt="Marble" />
<em>Natural Earth road dataset loaded in Marble. Notice the amount of detail, leading to screen clutter and the lack of any styling.</em></p>
<p>After this conversion, my actual work starts in which I have to build a tool using the Mable libraries.</p>
<ol>
<li>Which loads all these OSM files containg Natural Earth data.</li>
<li>Merges all the data in a single document.</li>
<li>Simplifies the data for a certain zoom level by reducing and filtering the number of nodes required for rendering.</li>
<li>Concatenates the polygons and polyline where it makes sense.</li>
<li>Splits the data into lots of different rectangular tiles.</li>
</ol>
<p>The core of the work will involve lots of mathematical problems (e.g. simplifying paths, concatenating paths, averaging paths, clipping paths to a rect, drawing coastlines etc.)</p>
<p>The goal is that by the end of GSoC, we will have a tool which cuts the natural earth dataset into tiles in
OSM file format which are properly styled and rendered by Marble for each tile level in the same way as they are
drawn for the bitmap tiles on openstreetmap.org</p>Hello everybody,
I am Akshat Tandon and I am a computer science undergraduate at
IIIT-H, India.
I have been selected in GSoC and this summer I will be working on Marble to improve its vector tiling and rendering capabilities.Building the Graph Coloring activity: Part 22016-01-14T18:29:00+00:002016-01-14T18:29:00+00:00http://droftware.github.io/Building-the-Graph-Colouring-activity<p><img src="http://droftware.github.io/assets/gcompris_logo.png" alt="Gcompris" /></p>
<p>The Graph Coloring activity is based on the famous NP Complete problem in which a graph ‘s nodes have to be colored in such a manner such that no two adjacent nodes have the same color.</p>
<p>My main aim for creating this activity was to prove the mentors that I can work in QML. I was able to code the working demo in QML but this code of mine was poorly built. My design was not extensible since I had not used JSON datasets for adding levels but had only hardcoded the graph information. I had copy pasted the code instead of using a model-views and repeaters. Also I had not kept the positions and sizes of elements relative to the screen ,which is very important for GCompris since it’s a cross platform application and is used on desktops, tablets and smartphones alike.</p>
<p><img src="http://droftware.github.io/assets/graph_coloring_icon.svg" alt="Gcompris" /></p>
<p>Days passed by and finally the mail came with the good news of my selection. Aruna Sankaranarayanan was assigned as my project mentor. However due to overload of projects and end semester exams I was not able to devote much time to GComrpis.
After everything academic related got over I took a break and it is after this break I again started the Graph Coloring activity. This time I jotted down the points which the mentors had told me after they had reviewed the code. I read the codes of other activities in GCompris so as to understand the correct way of representing and coding things as well as to pick and reuse certain things in my own Graph Coloring activity. I assumed that reading the codes of other will be an easy thing but it actually proved otherwise. The first time I picked a code to read and understand was a challenge but things eventually got better with me gaining a little speed.
Until now, any coding related task which I had done had not been “engineered” but was “hacked” to make the ends meet. In the open source world which I experienced though GCompris I found that there was much emphasis on quality and lousy work was just not permitted. Things are there for a specific well defined reason.
I submitted my second version after my disastrous first version. I had tried to rectify most of the things like extensibility, using model-views and repeaters instead of copy pasting. I thought that now my work on the Graph Coloring activity is over and that now I can start working on a new educational activity. This time also I was proved wrong. The mentors expected a high quality of code and told me rectify and correct a few things and thus began the iterative cycle of rectification and enhancements. I had to change things like representation of graphs in qml so as to make it more suitable for the application, and drawing the edges of the graphs using rectangles instead of canvas for enhanced performance.</p>
<p><img src="http://droftware.github.io/assets/gcompris_s2.png" alt="Gcompris" /></p>
<p>One thing which troubled me the most was to make the size and position of nodes of the graph relative to the screen. Each time I tried making some attribute relative something or the other got affected and I had to again tweak some things. However the mentors made me realize that this thing was very important for GCompris due to its cross platform nature and hence the game must be playable on any kind of standard resolution. Eventually I was able to make the size and position relative after repeated hit and trials.</p>
<p><img src="http://droftware.github.io/assets/gcompris_small_res.png" alt="Gcompris" /></p>
<p>Also I made the activity real time i.e. previously there was an Ok button which need to be pressed in order to get the coloring verified as to whether it is valid or not but now if you have colored the graph correctly you will automatically be notified and will be advanced to the next level. Apart from this animations were also added so as to help a child indicate in which adjacent nodes he has filed the same color.</p>
<p><img src="http://droftware.github.io/assets/gcompris_same_color.png" alt="Gcompris" /></p>
<p>One thing which I have observed is that GCompris tries to base its activities on good pedagogic principles. There were discussions in the mailing list regarding which features to add or remove so as to make a child better understand a certain concept.
A fine issue that was raised by Bruno was regarding color blind children. He told me to include shapes/letters apart from colors so that color blind children can easily label nodes with different shapes like stars,diamonds,hearts etc instead of coloring nodes. Hence an additional option was added so as to switch the mode from color to shapes and vice-versa. Eventually the shapes version was made the default version.
<img src="http://droftware.github.io/assets/gcompris_shapes.png" alt="Gcompris" />
Finally some graphs ranging from easy to difficult were added. Bruno helped me in this task by providing a script which parsed the information from svg <a href="https://en.wikipedia.org/wiki/Gallery_of_named_graphs">graphs provided by Wikipedia</a> into our application defined graph format.
<img src="http://droftware.github.io/assets/wiki_graphs.png" alt="Gcompris" />
Now since the activity is almost complete I am planning on starting a new activity. I have discussed some ideas with Aruna regarding a dictation activity and an algebra one. I am hoping to complete these before the end of Season of KDE.</p>Starting work on the SoK: Part 12016-01-14T10:18:00+00:002016-01-14T10:18:00+00:00http://droftware.github.io/Starting-work-on-the-SoK-Part-1<p>I got to know about Season of KDE through a facebook post and straight away I hopped onto the ideas page. Since I was a complete beginner in the open source domain I was hoping to find a project with code that is easy to understand and comprehend, has good documentation for newbie programmers and whose mentors are supportive and explain even the trivial matters.</p>
<p><img src="http://droftware.github.io/assets/gcompris_logo.png" alt="Gcomrpis " /></p>
<p>I came upon GCompris which is an educational software suite consisting of several activities and whose target user base is in the age group of 2 to 10 years. My task would involve adding new activities and/or porting existing activities from its GTK+ version to Qt Quick version.</p>
<p>I downloaded the code and played some of the activities. I joined their IRC channel and introduced myself to the project mentors and expressed my desire to participate in Season of KDE. The mentors Bruno Coudoin, and Johnny Jazeix were helpful and prompt in their replies. They told me that in order to contribute to GCompris I should know QML which is used to describe the user interface and Javascript for defining the logic.</p>
<p><img src="http://droftware.github.io/assets/gcompris_s1.png" alt="Gcomrpis " /></p>
<p>Now the catch was that I had no clue about QML. However I was determined to get selected since I had tried to contribute in some KDE organization a few months back but due to enormity and complexity of the code I had quit that time even before putting any efforts. The good thing about GCompris was their compartmentalized architecture in which all the 100+ activities were written entirely in QML and JS and were united together by a C++,Qt core which managed core functionalities. What also appealed me was that they had a new contributors page which had detailed instructions regarding how to setup git, create a new activity and sending pull requests. Since I was a newbie and had never worked with an open source organization ,an easy to follow architecture and a detailed beginners contribution page were very important.
Thus I made my decision to learn QML and apply in Season of KDE under GCompris. I started searching for some QML tutorials and got to know about the open source online book Qt5 Cadaques. Eccentric name but gem of a book ! With the help of this book I managed to build a working prototype of a Graph Coloring activity.</p>
<p>I would like to point out that GCompris has a script which generates all the boilerplate for the new activity which is required for integrating the new activity with the rest of the application as well as maintaining a unified look and feel. Misquoting Niel Armstring - this shell script might be small step for the mentors but it provided a big jump for me since now I did not have to fiddle with Cmake and other build related tools and could just concentrate on the main task of creating the activity, making the coding process much simpler.</p>I got to know about Season of KDE through a facebook post and straight away I hopped onto the ideas page. Since I was a complete beginner in the open source domain I was hoping to find a project with code that is easy to understand and comprehend, has good documentation for newbie programmers and whose mentors are supportive and explain even the trivial matters.