Saturday, 3 December 2016

Replacing OpenLayers

When OS released their Open Data I created a simple map that showed a few of their offerings as overlays on an OSM base layer oscompare.raggedred.net. At the time my maps were driven by OpenLayers. OpenLayers v2 works well but is a lot harder to use than LeafletJS. As soon as Leaflet arrived I started to use that and it has only got better as time has passed. There is OpenLayers v3 available I think, but I doubt I'll use it.

OpenLayers has a permalink option that changes the URL in the browser to include a query string describing the location of the centre of the map and the zoom level. This lets you bookmark or send a link that will open the map in the right place. Leaflet has a plugin to create a hash on the URL that does a similar thing.  It works well but there is a bit of a problem.

The OpenLayers permalink adds a 'layers=' bit to the URL. An example would be BOTFF. This means the first base layer is shown and not the second one, then the first overlay is visible but not the second or third. This great, because not only does the URL make map show the right place, but the overlay you wanted to show will also appear too. Leaflet's hash plugin doesn't do this and that is a problem for me.

I decided to replace the map showing the OS overlays using Leaflet. It was quick to do and works well, but I can't use it as the replacement for the OpenLayers map because Leaflet doesn't have a built-in way to show the layers that were selected. People link to my map, according to the logs, with the query string to select the place and layers. If I just replaced it with the Leaflet-based map their links would all fail. They couldn't create a new bookmark that includes the layers to show either. I decided to fix this.

I looked into the documentation for the latest version of Leaflet (1.0.2 at the time of writing). The documentation is very useful. If there are overlays on a map a layers control can be added that allows the overlays to be selected. There can also be more than one base layer that the control allows you to select.

I thought about a generic way to create a similar effect to the OpenLayers permalink. For a one-off map I can code up the layers to do as I need, but a generic version would be useful. I can get at the layers that a map is currently showing, but Leaflet's way of hiding a layer is to remove it from the map. The definition of that layer is then not accessible from the map object. I couldn't see how to find the list of layers including the ones not displayed. The layers control has a list of the layers it uses, which would be ideal, but I couldn't find any way to find a a reference to that from the map object. There doesn't seem to be any way to find any controls from the map object. A map method that returns a list of controls might be useful.

So far I have created a bespoke version of the Leaflet map I wanted. It will accept an OpenLayers-style URL, display the map and create a Leaflet-style URL. I'm thinking about how to duplicate the OpenLayers feature more generically. I have ideas ...

Monday, 14 November 2016

A Map in an App

Maps come in all shapes and sizes. I've made maps for a few different uses and since Openstreetmap came along there's always been some part of OSM in my maps. I offered to make a map for Clockenflap Music and Arts Festival in Hong Kong, so I expected to use OSM as the base. What I've ended up with uses the tools to create and display the map: JOSM, osm2pgsql, Tilemill, Mapnik and Leaflet, but none of OSM's greatest asset, its data.

Clockenflap, like most festivals, is largely a temporary venue. This year it's a new venue to Clockenflap on the Central Harbour-front Event Space. I looked at the area on OSM and how to map the temporary stuff that the event will put up. There's seven stages, food stalls, bars, attractions, art installations and all the usual support stuff like toilets, first aid and entrance and exit gates. That's a lot of stuff to add to OSM temporarily. Worst of all, when I started to look at it the site design was still not fixed. I quickly decided that it needed to be stored locally and not uploaded to OSM at all.

The map is embedded in an app, which I'm not writing. The idea is to run the map in a Webview with a JavaScript Leaflet-based display and it works well. The map has a base layer with all the services on it and layers of overlays with icons that can be clicked. In the apps (one iOS and one Android) the JavaScript in the Webview calls a native iOS or Android function when an icon is touched.  So instead of a web popup the app can respond with a dialog or a more detailed response. For example, if a stage icon is touched the app can show what acts are on the stage now and on next.  The reverse is true too. For example, if the user scrolls through the list of food suppliers in the app and chooses one, the map is called and shows where the food stall is on the site. I can't show the apps yet, but a working example of the map is here.

The festival team provided the icons and the look of the map. I think it looks very good. There are sponsors who need to see their logos and colour schemes and the map needs to reflect this, as well as being useful for getting around on a site that no one knows their way around because it is only a day or two old.

A few things are quite different from most of the maps I've made before. Driving the map from a phone screen is a challenge.  Trying to get specific icons on the map so they are big enough to read on smaller screen, but don't overlap too much is tricky. At the lowest zoom level (17) very little is shown by default. If all the overlays are show it is a jumble, there's just no choice.
Lots to fit on at low zoom if everything is turned on
Food stalls need room for their much longer real names
By the time we get to the highest zoom level (22) there's lots of space in most places. I needed that zoom level to comfortably resolve the food stalls, which are 3m x 3m, with an icon and room for their name. I needed to display a lot of food supplier names close together and the names needed to respond to a touch (or click). I used the Leaflet plugin Leaflet.Label for this. If I had used Leaflet 1.0, rather than 0.7, I think I could have used built-in tool tips instead.

There is an optical illusion when you zoom in on an icon, especially on a phone where you use pinch zoom. If you show the higher zoom level with the same size icon on it the icon seems to have shrunk. People expect pinch zoom to make things bigger, so I increase the size of the icons as each level zooms in. That is easy for the stuff on the base map, Tilemill makes that simple, but scaling the icons on overlays took a bit of code I'd not needed to use before. This involves resizing the icon and using setIcon() to use the newly sized icon in a marker. setIcon() is not documented in Leaflet 0.7, but works just the same.

The standard Leaflet layer control is much too small for a mobile phone. I tweaked the size and position with CSS. I also discovered that you don't need a base layer on the control and you can force it to always be open. Both are perfectly plainly described in the really good documentation of Leaflet, I'd just not read that bit before.

I control the markers on the map by describing them in geoJson files, one for each overlay. To position the many markers on the map I created a simple editor which uses the Leaflet.Draw plugin, which is really simple to use in code and produces a very useful editor.

I have enjoyed creating this map and working with creative people who have an absolute deadline to work to. If you are in Hong Kong on 25 - 27th November enjoy Clockenflap and enjoy using my map to find all the stuff Clockenflap has to offer.

Friday, 5 February 2016

Drains and meadows

I mapper jumped into mapping the rural areas of East Yorkshire more than a year ago but, sadly, seems to have stopped. This mapper has added farmland and woods in much more detail than anything before and has drawn the outlines of lots of buildings in villages too. There are a couple of things that I've decided to change in any areas that I take a look at now. That is meadows and drains.

Meadows are a special thing in Britain, with most having disappeared. They are not just areas of grassland but a very specific kind of management, involving no fertiliser, a careful cutting of the grass for hay more than once a year and some animal grazing, but not all year round. The timing of the hay cuts allows wild flowers to flourish as they have time to set seed before they are cut down. This makes a beautiful landscape that has all but vanished from the English countryside. The mapper has made the mistake that almost any green field in the aerial imagery is a meadow, so I'm checking as best I can on the ground and changing it to farmland unless it really is a meadow. The farmland is drawn with each field outlined which means I can try to add hedges when I see them. This too improves the quality of the rural mapping.

The other problem is that every field-side ditch has been mapped as a drain. This is further worsened by another mapper doing the same thing in Holderness, the flat land to the east of Hull. There are some waterways I would call a drain there, but most are ditches in my view. A ride out doesn't cover much ground to correct this and imagery is ambiguous at best. So there's lots to check and much will change.

The East Riding countryside is a great place to spend time; having an excuse to carefully explore and examine more of it in detail is no bad thing, so I'm not complaining.

Tuesday, 2 February 2016

Schools

People meet up in a few areas across Britain drawn together to share views and ideas about OSM. One of these is known as Mappa Mercia, meeting in the old Kingdom of Mercia, roughly the Midlands. They started a quarterly series of projects which encourages people to focus on improving specific areas, such as nature reserves. This quarter it is schools.

People across Britain have taken to the suggestion of checking schools and improving the quality of recording schools. This is a simple idea that will greatly improve the quality of data in OSM, something that the quarterly projects have always aimed at. It's a great idea and something I support.

There is a list of UK schools available as open data which Robert has used to show a map of how well the schools in the list are represented in OSM. It uses postcode areas as that is how the open data is structured. This is a very good indicator of areas that still need work.

This is all useful, but then I think the project took a wrong turn. People were encouraged to add a hashtag to changeset comments. This has then been used to produce a list of people who have contributed the most changes to the project. I don't like this. OSM is not a competition and turning it into one encourages edits just to climb the list (I'm not accusing anyone of that, just that the pressure is created). We have seen that with the list of anomalies published between OS Locator road names and OSM road names. To push an area up the list people just use the OS Locator names without checking. I know that OS Locator contains errors and there is a mechanism to record these, but many areas have very few recorded errors, so erroneous OS data is now in OSM.

I like the idea of focussing on improving specific things in OSM. It should encourage surveying. I would prefer it is not be seen as a competition.

Friday, 27 November 2015

When is a town not a town?

I live a few miles away from a rapidly growing settlement called Brough. I see it mentioned from time to time as a town in the local press, but I've always thought of it as a large village. It has been a settlement for a long time, it was a town, Peturia, when Rome ruled this area.

The idea of not knowing whether a place in England is a town or a village seems easy to resolve: does it have a town council or a parish council? Well in the case of Brough it is not quite so easy. The local council is a town council, but it covers both Elloughton and Brough and is called Elloughton-cum-Brough Town Council. It is not that uncommon to find a civil parish or town council with multiple settlements within its bounds, but the council is a town council, surely one of the settlements needs to be a town.

It is clear to me there are two settlements here. Road signs show both places sometimes in different directions, OS Locator shows the street names as streetname : Brough : Elloughton-cum-Brough and streetname : Elloughton : Elloughton-cum-Brough.

OS Open Names shows both places as populatedPlace: village. So is it possible to have a town council presiding over an area with two villages and no towns in it?

I've asked the Local Authority, who, after all, provide the data for OS Locator and OS Open Names, what they think.

The answer is a political hot potato at the moment, as Elloughton-cum-Brough town council have recently decided to spend £4000 on a mayoral-style chain for the head of the council. It seems he can legitimately call himself mayor after the town council made the appropriate change in 2011. That's a lot of money to spend on what some call trinkets at a time when there are council cuts elsewhere.

Edit:
The clerk to the Elloughton-cum-Brough has contacted me and kindly given me information about the process that resulted in the parish council becoming a town council. She also confirmed that both Brough and Elloughton are villages. That means that the town council area has two villages and no towns in it.

Monday, 28 September 2015

Extracting building heights from LIDAR

The UK Environment Agency have released some LIDAR data in Digital Elevation Model (DEM) format. It includes Digital Terrain Model (DTM) data and Digital Surface Model data. The DSM data includes buildings and trees while the DTM is processed to remove these so the underlying terrain is visible. Tim Waters asked if you subtract the DTM from the DSM would you be left with just building and tree heights? I'd started to look at this and it turns out building heights are extractable in this way.

I have written a script to do the subtraction and create both the difference file and an SQL file to load the data into a postgresql table. I created a hill shading image from one of the difference files to see what features have been stripped from the DTM data. Here's a jpg version of it:


You can see that the buildings and some other features are all well defined. Phil Endecott suggested using the DSM data to create building outlines which could be traced in OSM. His images look good. I would suggest starting his process with this difference data as all the terrain detail has been removed so it may be even better.

Once the SQL file of height data has been loaded into a Postgresql database, which has the PostGIS extension installed, we can then do some queries on it. I selected some OSM building polygons in the range of the loaded data and found which of the height points fell within each polygon. The highest of these is the highest point of the building above the surrounding terrain. I've written a script as a place-holder to extract all the heights for a rectangular area. Someone could extend this to make a file to upload to OSM to add the heights for every building in the defined area. I feel this is clearly an import and so the usual OSM import process needs to be gone through before the import takes place.


I think there is real merit in using this data to extract building heights, which are needed for the 3D images of city buildings.

The two scripts are available here: http://www.raggedred.net/shared/heights.zip The first (s-t.py) needs matching DSM and DTM files and outputs a difference file and the SQL to load into a database table. This will work for any of the resolutions published by the EA. The second, much less polished, script (match.py) defines a rectangle, extracts the OSM buildings for that rectangle and then finds the height data for each building. I wrote it as a script so it can be extended to create a data file for processing or uploading or so overlay tiles could be made from it. I loaded some OSM data with osm2pgsql (which would normally be used for rendering) and added the table for the heights data to the database. The SQL for the table is:

CREATE TABLE eadata
(
  hid serial NOT NULL,
  height double precision,
  locn geometry(Point,27700),
  CONSTRAINT eaheight_prim PRIMARY KEY (hid)
)
WITH (
  OIDS=FALSE
);

CREATE INDEX eadata_index
  ON eadata
  USING gist
  (locn);
The output SQL data can be loaded into this with the command:
psql -f
where and are whatever you used. The table name eadata and field names are hard-coded in s-t.py. 

It is important to say that I would not use a rendering database to create the upload data from as some fields will be missing. osm2pgsql is a lossy process. You can use the OSM ID to extract a current version from the API or from Overpass to add the height data to. I used the rendering data for convenience as I already had it available and to satisfy myself that the process works.  

I hope this is useful to someone. Please feel free to ask for more information if I've not made anything clear.

Sunday, 20 September 2015

More LIDAR goodness

I looked into LIDAR data from the UK Environment Agency some weeks ago. I needed it help a local group who are investigating flood mitigation options. The data was listed as being about £26,500 but we got a 100% discount if we used it for restricted, research use, so we could afford it! A few weeks after I'd used the height data for the group I got an email from the Environment Agency. They said the data was being made available as Open Data under the Open Government Licence. So now I could use it for any other purposes at no cost. You can get the data from http://environment.data.gov.uk/ds/survey

I decided to make a detailed relief map of part of the area close to home. The data doesn't cover the whole country, only parts that are deemed at risk of flooding. All of Hull and the river Hull catchment area are included in this. I've only looked at my local area so other areas may vary.

The data is downloaded as Ordnance Survey 10km grid tiles. There are 2m, 1m, 50cm and 25cm options and digital terrain model and digital surface model options too, so let's look at these options, but first a bit about LIDAR.

LIDAR is a technology that uses laser light to measure a distance repeatedly over an area to create a 3D model of an area. If the LIDAR transceiver is mounted at a fixed point it can pan around to record a very detailed image in 3D of everything that can be seen from that point. It works very well in this way inside a building or a cave to make a very accurate model. The US Space Shuttle flew a mission to use a variant of LIDAR to record the height of the surface of the Earth from space. This is available as SRTM.

More recently LIDAR equipment has been flown in aircraft. The difficulty of making useful measurements from an aircraft should not be underestimated. The only data LIDAR returns is distance to the target, so knowing PRECISELY where the aircraft is in 3D is the real problem. GPS is hopeless at altitude measurement and scarcely good enough for lateral location, barometric height measurements vary over time and location and inertial dead-reckoning accuracy falls off with time. A combination of all of these plus post-processing can result in useful data.

The Environment Agency LIDAR distance options specify the distance between the sample points, the 2m option having less detail than the 25cm option. The area that these options cover varies with the highest detail covering the smallest area. I chose the 50cm option as it covered the area I wanted at the highest level of detail. The detail does make for larger datasets and more processing needed to do anything with it.

Clearly the LIDAR measures the distance to the first object it encounters from the aircraft, so it measures tree tops, building roofs and even vehicles. This is known as the digital surface model. This is often a composite from multiple images, as this data is, to compensate for location inaccuracies and to help remove things like vehicles. To get a useful model of the real landscape, without trees and buildings, the data is post processed to create the digital terrain model. This is the data I have used.

The OGL data was different from the data the Environment Agency originally supplied. The original data was in smaller grid squares and the height was rounded to the nearest centimetre. The OGL data is in bigger squares which makes it a bit easier to process but seems to use 18 decimal places of a metre, which is smaller than the diameter of an atom.

I wanted to create a relief map and make contours from the data and, not for the first time, GDAL had the tools. The data uses the UK Ordnance Survey projection, known as OSGB36 or ESPG:27700, so to use any OSM data with it I would need to reproject to WGS84 or EPSG:4326.

To make a relief map I used gdaldem with the hillshade option on each of the datafiles. These need to be joined together to make a larger image, so the option -compute-edges is also needed. The complete command is:
gdaldem hillshade -compute_edges infile relieftiff/outfile.tif
The output is geoTIFF files which can be merged into a single (large) geoTIFF with the command
gdal_merge.py -o big.tif *.tif
This creates a geoTIFF file which has the image of the relief in a TIFF image and also has the locations of the edges in the original OS projection.

The next step is to use gdalwarp to reproject the large tiff file to one in the WGS84 projection. The command describes the source and target projection and filenames. There are significant missing pieces in the large TIFF as the available data was not rectangular. The -srcnodata 0 and -dstalpha makes missing data transparent rather than black.
gdalwarp -s_srs epsg:27700 -t_srs epsg:4326 -srcnodata 0 -dstalpha big.tif bigr.tif
The new TIFF file is what we want to see, but now it needs turning into tiles to be displayed on a slippy map. I decided that zoom level 13 to 18 would give a useful display. To make these tiles I used gdal2tiles.py, specifying the reprojected TIFF image, the zoom levels and the folder to put the tiles into.
gdal2tiles.py -z13-19 bigrt.tif tiles
This makes a set of tiles in the TMS format in the specified folder, in this case tiles.

Another way to visualise the LIDAR data is contours. I decided to create a set of overlay tiles that are transparent except for the contours. These can have a different density of contours at each zoom level. I chose the smallest contour step to be shown at the highest zoom level to be 0.2 metres. The GDAL tool for the job is gdal_contour which makes a shapefile with a linestring for each contour. The command is
gdal_contour -i 0.2 -a height infile outshapefile.shp
The resulting shapefile needs to be reprojected to WGA84. The tool to reproject shapefiles is ogr2ogr
ogr2ogr -t_srs epsg:4326 -s_srs epsg:27700 outshape.shp new.shp
I decided to use Mapnik to make the contour overlay tiles. Mapnik can use shapefiles but specifying the long list of shapefiles created above would be a problem so I loaded the shapefiles into a postgresql table in a database with PostGIS enabled. Postgresql comes with shp2pgsql to do this:
shp2pgsql -a -g way new.shp eacontours > new.sql
This makes SQL to load the shapefile into eacontours table, putting the geometry in the field called way. To load this into a database called ukcontours which already has the postgis extension installed the command is
psql -d ukcontours -f new.sql
I then designed the overlay with Tilemill to create the transparent tiles with more contours at higher zoom levels.

You can see the results at http://relief.raggedred.net. I added a water overlay and a roads overlay (thanks to MapQuest for the roads) to help position the relief imagery.