Visualization of Sentinel-2 GeoJSON Data

Project Goal

EOX is our partner in the Minerva project, and they specialize in geospatial databases. EOX and most of their projects are very keen on supporting open standards such as OGC and so on. In the course of their collaboration with ESA in the Copernicus programme they built a public interface to Earth observation data collected by the Sentinel-2 orbiter available as a REST service (trythis) Nowadays, everything is a REST service, which simple means that certain parameters in a URL are your query parameters and you get data back via an http request, typically an xml or json file. This URL provides a REST service adhereng to open search standards http://ows.eox.at/evo-odas/eoxserver/opensearch/collections/sentinel2/json/. Our goal is to build a small mockup with Aardvark.Media, which queries this REST service and interprets and visualizes the data provided in the form of GeoJSON.

Preparation

First, we spend some time on planning and make a short roadmap consisting of milestone. Each milestone is a runnable prototype, which in the best case visualizes something. In the most general way, our application will be a concatenation of mappings, transforming our input data a couple of times until we see pixels on the screen. Most of the apps we program can be characterized this way, some give it a fancy name like "the data flow" pattern, but that's a 100 dollar name for a 1 dollar concept.

In the functional world, transformations are functions, and the things we transform are types. A first sketch for this program, for the whole program btw, may look like this: .json -> Model -> ISg. From that we can create a module that will have the following functions:

module GeoJson =

  type Model = {
    data : int
  }

  let load (jsonFile : string) : Model = failwith "not implemented"

  let view (m : Model) : ISg = failwith "not implemented"

And these two transformations are already our two milestones

• Milestone A: Parse and load json data into model
• Milestone B: Transform model to Sgs and view data

This is the initial step to every new feature we implement: Take a step back, think about the types (we didn't do that really), think about the transformations and then build a small media scratch, which is an empty husk with a lot of failwiths and dummy types.

To build a media scratch do the following:

• clone media and checkout/switch to master
• open media with vs, locate the scratch folder and open new.fsx
• Ctrl+A, Alt+Enter sends the code to fsharp interactive
• Type in a name for your scratch in the interactive console
• Let vs reload the project file
• Commit and push empty scratch project to media master

When starting the Scratch project an Aardvark will greet you!

The script 'new.fsx' is an fsharp script, which parses the current solution file, determines the index number of the new scratch project and appends it to the currently open media solution, hence the reload is necessary.

Loading the Data

Let's have a look at what data we want to load and open the EOX link in Firefox to get the following, human readable json printout:

Next, we look for an fsharp library that can "load" a json file. It is difficult to answer what a suitable library is. As a starting point we can use a type provider, which would automatically infer a domain model for us from a sample json file.

Integrating a JSON type provider and parsing the data

We look at the JSON type provider documentation and try to add the following to our GeoJson module:

namespace GeoJsonViewer

open Aardvark.Base
open FSharp.Data
open FSharp.Data.JsonExtensions

module GeoJSON =

  type EoxModel = JsonProvider<"..\..\..\data\eox.json">

  let load (jsonFile : string) : Model = 
    let eox = EoxModel.Load(jsonFile)
    Log.line "%A %A %A" eox.Bbox eox.Type eox.Features
    { value = 0 }
  ...

In this case JSON data cannot be found, we need to reference the FSharp.Data package first. We need to perform the following steps to integrate a new package:

• lookup the newest package version on nuget, in our case this is 3.0.0 on link
• add the package and version to the paket.dependencies file, nuget FSharp.Data ~> 3.0.0
• add the package to the paket.references file in our project FSharp.Data
• close visual studio and call .\.paket\paket install which generates an updated paket.lock file and updates the references in the project files. This is important: we do NOT need to add references via visual studio, paket does all that

Reopening Aardvark.Media we discover that JsonProvider can actually be found. The type provider needs a sample file at compile time to build a type before we compile our code. Therefore we download the json file from the REST service and put it into the data folder of Aardvark.Media, hence the path "..\..\..\data\eox.json". Next step we want to call our code. A nice variant would be to use fsharp interactive, but I always have difficulties to set that up properly (Harald?). Using CTRL+F5 really starts up fast in case we just want to look at the result and don't want to debug. Calling the above code can look like this:

GeoJSON.load @"..\..\..\data\eox.json" |> ignore
Console.Read() |> ignore
Environment.Exit(0)

//console output
0: [|7.386474M; -19.078916M; 50.147561M; 49.652723M|] "FeatureCollection" [|{
  "type": "Feature",
  "properties": {
    "id": "S2A_MSIL1C_20170501T100031_N0205_R122_T33TUN_20170501T100029.SAFE",
    "begin_time": "2017-05-01T10:00:31.026000Z",
    "end_time": "2017-05-01T10:00:31.026000Z"
  },
  "bbox": [
    12.777443,
    46.842299,
    13.816648,
    47.847371
  ],
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [
          12.777443,
          46.842299

We have reached the first step to MilestoneA, we can parse the json from a local file and output it onto the console. We wrote a very small piece of code and immediatly verified if it does what we expect it to do. This is a very crude test driven approach, where we actually don't write any tests :D

Building a Domainmodel and actually loading the data

The json type provider only helps us in parsing and accessing the data but provides a rather inconvenient type for general use. To create a domain model we can actually use for our app we inspect the fields of the inferred type, or the original json. We only look at the first level and the details we postpone. Our preliminary domain model may look like this:

type Typus =
  | FeatureCollection  

type Feature = int

[<DomainType>]
type Model =
  {
    boundingBox : Box2d
    typus       : Typus
    feature     : list<Feature>
  }

initial =
    {
       boundingBox = Box2d.Invalid
       typus       = Typus.Feature
       feature     = List.empty
    }

Now we go into more detail when parsing the json data and how we get it into our own domain model. Once again, we scratch up some outlines and postpone the details to later. Basically we construct a parser hierarchy top down, but just defining transformations from one type to the other.

let parseBoundingBox(bbox : array<decimal>) : Box2d = failwith ""

let parseRoot (model : EoxModel.Root) : Model = 
  {
    boundingBox = root.Bbox     |> parseBoundingBox
    typus       = failwith ""
    features    = failwith ""
  }

let load (jsonFile : string) : Model = 
    EoxModel.Load(jsonFile) |> parseRoot

Notice how failwith can be used at any position in the code and still leaves your program compilable. Be careful however with empty string failwiths, since they might be difficult to find later on in a bigger program with error suppression, or when you build packages with such mechanism which then crash without any information. Continuing in this fashion, we arrive at the following:

module GeoJSON =

  type EoxModel = JsonProvider<"../../../data/eox.json">

  let parseBoundingBox (bb : array<decimal>) : Box2d = failwith ""

  let parseTypus (typus : string) : Typus = failwith ""

  let parseFeature (feature : EoxModel.Feature) : Feature = failwith ""

  let parseFeatures (features : array<EoxModel.Feature>) : list<Feature> =
    features |> Array.toList |> List.map parseFeature

  let parseRoot(root : EoxModel.Root) : Model = 
    {
      boundingBox = root.Bbox     |> parseBoundingBox
      typus       = root.Type     |> parseTypus
      features    = root.Features |> parseFeatures
    }

  let load (jsonFile : string) : Model =
    EoxModel.Load(jsonFile) |> parseRoot

In the final step we actually fill the parsers with implementations, one by one, and reuse parsers we already implemented. Simultaneously we need to update our domain model by new types and union cases and so on. Showing this step by step is not necessary, I guess you already got a good grip of this handson, early feedback approach, so our final domain model and parser look like this:

final Domainmodel

type Typus =
  | FeatureCollection
  | Feature
  | Polygon

type FeatureId = FeatureId of string

type Properties =
  {
    id        : FeatureId
    beginTime : DateTime
    endTime   : DateTime
  }

type Geometry =
  {
    typus       : Typus
    coordinates : list<V2d>
  }

type Feature =
  {
    typus       : Typus
    properties  : Properties
    boundingBox : Box2d
    geometry    : Geometry
  }

[<DomainType>]
type Model =
  {
    typus       : Typus
    boundingBox : Box2d
    features    : list<Feature>
  }

Final Parser

module GeoJSON =

  type EoxModel = JsonProvider<"../../../data/eox.json">

  let parseBoundingBox (bb : array<decimal>) : Box2d =

    if bb.Length <> 4 then failwith "invalid bounding box of size other than 4"

    let minLat = float bb.[0]
    let minLon = float bb.[1]
    let maxLat = float bb.[2]
    let maxLon = float bb.[3]

    Box2d(minLon, minLat, maxLon, maxLat)

  let parseTypus (typus : string) : Typus =
    match typus.ToLowerInvariant() with
      | "featurecollection" -> Typus.FeatureCollection
      | "feature"           -> Typus.Feature
      | "polygon"           -> Typus.Polygon
      | s -> s |> sprintf "string %A unknown" |> failwith

  let parseSingleCoord (c : array<decimal>) : V2d =
    if c.Length <> 2 then failwith "invalid coordinate of size other than 2"
    V2d(float c.[0], float c.[1])

  let parseCoordinates (coordinateSet : array<array<array<decimal>>>) : list<V2d> =
    [
      for set in coordinateSet do
        for c in set do
          yield c |> parseSingleCoord
    ]

  let parseGeometry (geometry : EoxModel.Geometry) : GeoJsonViewer.Geometry = 
    {
      typus       = geometry.Type        |> parseTypus
      coordinates = geometry.Coordinates |> parseCoordinates
    }

  let parseProperties (properties : EoxModel.Properties) : Properties =
    {
      id        = properties.Id |> FeatureId
      beginTime = properties.BeginTime.DateTime
      endTime   = properties.EndTime.DateTime
    }

  let parseFeature (feature : EoxModel.Feature) : Feature =
    {
      typus       = feature.Type       |> parseTypus
      boundingBox = feature.Bbox       |> parseBoundingBox
      properties  = feature.Properties |> parseProperties
      geometry    = feature.Geometry   |> parseGeometry
    }

  let parseFeatures (features : array<EoxModel.Feature>) : list<Feature> =
    features |> Array.toList |> List.map parseFeature

  let parseRoot(root : EoxModel.Root) : Model =
    {
      typus       = root.Type     |> parseTypus
      boundingBox = root.Bbox     |> parseBoundingBox
      features    = root.Features |> parseFeatures
    }

  let load (jsonFile : string) : Model =
    EoxModel.Load(jsonFile) |> parseRoot  

Showing the Data

So far our data visualization only took place in the console, so we want to show the different geojson features in different ways. We can speak of features, as they are in the domainmodel, but in a more abstract way we may call one single element of our data an entity. Entities live in our domainmodel and view functions, i.e. transformations from model to sth. viewable, are rules on how we want to visualize our data. So in this chapter we will provide multiple view functions: (1) a html list view, (2) a map-like svg view, and (3) a 3D view consisting of ISgs.

From this enumeration the transformations become apparent, we need view functions with the following signatures:

Feature -> DomNode  //html list item
Feature -> DomNode  //svg
Feature -> Sg       //scenegraph

An Html List View

let semui =
  [
    { kind = Stylesheet; name = "semui"; url = "https://cdn.jsdelivr.net/semantic-ui/2.2.6/semantic.min.css" }
    { kind = Script;     name = "semui"; url = "https://cdn.jsdelivr.net/semantic-ui/2.2.6/semantic.min.js" }
  ]

let view (model : MModel) =

  let content =
    alist {
      for f in model.data.features do
        let (FeatureId id) = f.properties.id
        let item =
          div [clazz "item"][
            i [clazz "large map pin middle aligned icon"] []
            div [clazz "content"] [
              a [clazz "header"][text "Feature"]
              div [clazz "description"] [text id]
            ]
          ]
        yield item
    }

  require (semui)(
    body [] [
      Incremental.div ([clazz "ui relaxed divided list"] |> AttributeMap.ofList) content
    ]
  )

• Aardvark.Media DomNodes
• Transisition Model to MModel
• Incremental DomNodes

Docking Interlude

Curerntly, we can only display one view function at a time since we only have one page and we can only have one html body. So before we write another view function we integrate Aardvark.Media.Pages which also allows us to have dockable and resizeable subwindows. The concept of pages is rather simple: basically we extend our view function to support multiple viewfunctions depending on an identifier. Behind the scenes, this identifier is a subdomain of our apps webservice.

//dockconfig
initial =
  {
    data = GeoJSON.load @"..\..\..\data\eox.json" 
    docking =
      config {
          content (
              horizontal 10.0 [
                  element { id "map";  title "2D Overview"; weight 3; isCloseable false }
                  element { id "list"; title "Features";    weight 2; isCloseable false }
              ]
          )
          appName "GeoJSON"
          useCachedConfig false
      }
  }

//paginated view function
let view (model : MModel) =

  let content = [...]

  page (fun request ->
    match Map.tryFind "page" request.queryParams with
      | Some "list" ->
        require (semui)(
          body [ style "width: 100%; height:100%; background: transparent; overflow: hidden"] [
            Incremental.div ([clazz "ui very compact stackable inverted relaxed divided list"] |> AttributeMap.ofList) content
          ]
        )
      | Some other ->
        let msg = sprintf "Unknown page: %A" other
        body [] [
            div [style "color: white; font-size: large; background-color: red; width: 100%; height: 100%"] [text msg]
        ]
      | None ->
        model.docking
          |> docking [
            style "width:100%;height:100%;"
            onLayoutChanged UpdateConfig
          ]
  )

An Svg Map View

We can understand the svg view just as another transformation function, which in this case transforms our features into svg objects. We will not discuss the whole source code or all steps at this place, since the snippets become larger and larger. Instead, we will focus on the most important parts and describe how they relate to Aardvark.Media features. Similar to html elements we can create svg nodes, so our view function looks quite similar to the previous one

let svg = 

  let canvasSize = V2d(640.0, 480.0)

  let svgAttr = 
    amap {
      //let! bb = bb
      yield "width" ==> sprintf "%f" canvasSize.X
      yield "height" ==> sprintf "%f" canvasSize.Y
      yield "viewBox" ==> sprintf ("%f %f %f %f") 0.0 (-canvasSize.Y) canvasSize.X canvasSize.Y
      yield clazz "svgRoot"
      yield style "border: 2px dashed black"

    } |>  AttributeMap.ofAMap

  let content =
    alist {
      let! bb = model.data.boundingBox
      yield bb |> svgDrawBoundingBoxNorm svgGlobalBBStyle bb canvasSize

      for f in model.data.features do
        yield f |> svgDrawFeature bb canvasSize
    }

  Incremental.Svg.svg svgAttr content

drawing a single svg rect

let svgBBStyle =
  [
    "fill" ==> "blue"
    "stroke" ==> "darkblue"
    "stroke-width" ==> "1"  
    "fill-opacity" ==> "0.1"
  ]

let svgDrawBoundingBox attributes (box : Box2d) =  
  Svg.rect <| attributes @ [
        "x" ==> sprintf "%f"      box.Min.X
        "y" ==> sprintf "%f"      (-box.Min.Y - box.SizeY) //svg coordinate system flip
        "width" ==> sprintf  "%f" box.SizeX
        "height" ==> sprintf "%f" box.SizeY
    ]
...
feature.boundingBox |> svgDrawBoundingBox svgBBStyle

Linked Hovering

Now we have two different visulizations, an svg map view and an html list view, showing the same data entities. Now to show which box belongs to which list entry we write a simple hover selection.

drawing a single svg rect with hover selection:

type Action =
  | Select of Id

let drawFeature (feature : Feature) =
  let interaction =
    onMouseEnter (fun _ ->
      feature.properties.id |> Select)

  let attributes = interaction :: svgBBStyle
  feature.boundingBox |> svgDrawBoundingBox attributes

Adding the Feature Geometries to the map view

A 3D Sphere Projection