Using Spatial ETL in a Multi-Vendor GIS Environment (Part 2)

by Dennis Beck on March 17, 2014

As described in my previous post, GIS architectures are evolving from environments where all the functionality resides in the corporate GIS platform to new architectures where external applications are providing capabilities that have historically been performed within the corporate GIS.  There are two general directions that these new architectures are taking: 1) a “Geo-Centric” architecture, and 2) a “Database-Centric” architecture.  This post describes the geo-centric architecture and how it is being used today.

A geo-centric architecture is a somewhat subtle change from environments where organizations have one, common, enterprise GIS.  In this environment, the GIS is still the common source of truth for the organization.  The GIS is responsible for several key roles, including:

  • Managing the vendor’s data model
  • Representing a common network model, which is particularly important for utility and telecommunications organizations
  • Managing the associated metadata structures, and
  • Conflict resolution to support multi-user editing within the corporate GIS

External products that perform graphical functions, such as CAD-based design, or geospatial visualization can be integrated with the GIS database in multiple ways, but the following are common examples:

Loosely coupled approach – in this case, an ETL (extract-transform-load) product, such as FME from Safe Software (www.safe.com) can be used to perform a translation out of the enterprise GIS database to provide source data to support new application requirements.

The advantages of this approach are that it can be a simple, relatively quick and low cost means for leveraging data within an existing corporate GIS environment.  The primary issue with this approach is that once the data is out of the corporate GIS it is no longer under the management of the GIS platform.  There are workarounds for this, based on concepts such as a check-in and check-out, but this loose disassociation can lead to data integrity issues if not carefully managed.

Tightly coupled approach – Another approach involves more tightly coupling the external application to the GIS.  An example of this is through the use of the OSGeo Feature Data Objects (FDO) API.  FDO allows graphic applications to access geospatial data from the source database directly where it is stored.  Each FDO source has a “Provider” that is accessed by the appropriate client.  More about FDO is provided at the OSGeo FDO landing page, http://fdo.osgeo.org/.  An example is shown in the diagram below.

The FDO approach offers an elegant way of accessing geospatial data to make it available to external applications.  In some cases it can even be cached in the external environment where it can be utilized to support visualization and editing operations.  There are also challenges with this approach however; not all vendors support these open standards; and editing, while available, must still be well-managed to ensure that the source database receives valid updates.

FDO natively accessing data from multiple spatial sources

 

The geo-centric architecture is something that organizations have been adopting for some time.  For some organizations it is a planned means of accessing their GIS data.  For many however, it just happens, usually on an unstructured basis.  As mentioned in an earlier post, organizations are utilizing their corporate spatial information in best-of-breed tools, mash-ups and other forms of ad hoc analysis to support important business decision making.  The challenge is to ensure that this valuable, enhanced geospatial information does not wind up lost on an individual’s hard drive.  Rather, it needs to be made available to the broader organization to enable advanced decision support, while retaining proper data integrity.

Our next post will discuss database centric spatial architectures and how they are being used in organizations that need to manage geospatial information at an enterprise level.

Read More
Dennis BeckUsing Spatial ETL in a Multi-Vendor GIS Environment (Part 2)

Using Spatial ETL in a Multi-Vendor GIS Environment (Part 1)

by Dennis Beck on January 12, 2014

I was involved with my first GIS migration project in 1987. The client, a large electric utility, had a business requirement to merge three different organizations that were all using different geospatial systems. One system was home-grown, the other was from a vendor that was exiting the geospatial market and the third product was as an established player in the market. For the next 15 to 20 years I was involved, in one way or the other with various GIS consolidation projects. The objectives were always similar:

  • Create a common enterprise database to serve the long-term needs of the organization
  • Implement processes and applications around data creation and maintenance to allow the system to be kept up-to-date via normal business processes
  • Get the organization on one common platform to reduce the number of systems that need to be maintained, thus reducing overall cost of ownership

As time has marched on, a new phenomenon has emerged. These organizations that were so focused on having a common geospatial platform are now branching out and using multiple GIS and CAD products to create, maintain, manage and analyze their geospatial information. We can ask the question, “What went wrong?” “Why are all these different systems proliferating throughout the organization?” Actually, there are a lot of things that are “going right”. Geospatial technologies, as they continue to become mainstream applications, are getting incorporated into more and more business processes within organizations. The figure below shows several different consumers of spatial information at a typical utility.

Today organizations are buying specialty, packaged applications to address specific business needs. Advanced CAD-based design tools are providing more advanced editing and engineering environments for design professionals. And the different user communities have vastly different requirements in terms of usability and application functionality.

What are the technology shifts that are driving these changes?

  1. While systems are much more advanced, they are not as complex, or expensive to maintain as they used to be. When I started in the industry a standard workstation was over $100,000 USD. Now powerful GIS applications can run on your phone and they certainly don’t take a dedicated professional to maintain.
  2. Computer performance has not only reduced system costs, but it has changed the paradigm for accessing spatial information. It is now possible to dynamically translate large amounts of spatial data within an interactive application. These operations used to take days (sometimes weeks) to process, using systems of high-powered mainframe computers.
  3. New tools and integration standards are facilitating more open, interoperable systems.

While there is great opportunity with these new systems, there is also a risk that the enterprise data management for these large, integrated systems can be compromised. This series of posts will describe how new architectures are evolving to support these multi-user, interoperable environments. A key component of these solutions is the use of spatially enabled ETL (Extract-Transform-Load) technologies that support advanced data sharing requirements for enterprise users.

 

Read More
Dennis BeckUsing Spatial ETL in a Multi-Vendor GIS Environment (Part 1)