1. International geocoding requires conversion from vastly differing address formats across many different countries making it difficult to obtain a one size fits all solution.  This requires us to implement specific solutions for different countries or areas of the world.
2. Of the well known and widely available international geocoding solutions most are not free or prevent the commercial use within a third party product.
3. Many international geocoding solutions are quite costly and can cost hundreds or thousands of dollars to geocode just a few thousand addresses.

What can I do now if I have International data that I want to use in SpatialKey?

If your existing data does not have latitude and longitude and is outside of the United States there are several third party solutions that you can use to geocode your data before importing into SpatialKey.  The USC GIS research laboratory provides a comprehensive listing of free and paid Geocoding options. There are very few free services that offer bulk geocoding for international addresses but you can utilize one of the free services listed at USC GIS research laboratory and script your own solution for bulk geocoding. Using a combination of a third party geocoding solution  and our data import API you can automate  management of your data within SpatialKey.

One solution that provides a free bulk geocoding for European data with up to 5000 addresses per day is http://www.batchgeocode.com. BatchGeocode uses the Yahoo API’s and supports both United States and European addresses.

Will SpatialKey provide International geocoding in the future?

We will continue to search for a low cost, easy to use International geocoding solution that can be integrated with SpatialKey.  If you have an immediate need for international geocoding within SpatialKey we can custom develop a solution that integrates with a third party geocoder. Contact our sales team if you are interested in a custom solution.

Here are the basic steps to get started with the DIAPI:

1. Generate an XML file describing the dataset and a CSV containing the actual data. See Creating Data Import XML and CSV files for more details.
2. Use the HTTP services to authenticate and upload these assets. See the Developer Guide and DIAPI Documentation for more details.

To help you get started we have provided samples in both Java and ColdFusion. Additionally we provide an example application that you can download, customize and deploy called the SpatialKey Data Poller.

In just a few minutes, SalesForce users can gain a unique view into location patters related to their leads, opportunities, contacts, or accounts.  We really enjoyed showing this example to attendees at the recent Web 2.0 Expo in San Francisco.  Several people mentioned that they had been looking for a way to map their SalesForce data for quite some time.  They were amazed at how easy this was to do, and at the interactivity of the reports.  Others had never thought to visualize their CRM information this way.  It was fun to watch the “ah ha” moment as people started to think of how this geographic intelligence could be applied to their sales and marketing efforts.

We gave away lots of beta invites at the show, so we are looking forward to hearing feedback from people visualizing their SalesForce data with SpatialKey. We’re especially interested to hear feedback about what other features or workflow users might like to see related to CRM visualizations. At the top of the list so far is the ability to associate your CRM data with third party demographics data, so we’re giving that some thought…

Here’s the first video from the article.

Parking Ticket Map from SFGate.com

The San Francisco Chronicle’s website, SFGate.com, has a nice map showing the top locations in San Francisco where parking citations are issued. The dataset includes individual locations where 100 or more citations were issued, so it’s a map of the single places you’re most likely to get ticketed (but note that it doesn’t include the full dataset, only the top 576 locations). They’ve created a map that uses the Google Maps API and they overlay their own custom markers that use graduated circles to represent the number of tickets issued at any given location. The size of the circle indicates how many tickets were issued, and each unique location has one circle centered on the location.

But there are a few problems with this visualization. The two most obvious things that stand out are the difficulty in understanding the density of many map markers all overlapping one another, which is seen in the north-east area of the city (downtown), and the second issue is the fact that the one huge marker at the southern edge of the city makes every other marker look tiny and unimportant. A single glance at this map would lead me to conclude that there must be more parking citations issued in the southern area of the city than in the other areas. But that’s the wrong conclusion to draw.

The problem with overlapping markers

One of the big issues that we see with maps that contain a lot of data points is that the kind of markers that are typically used in online maps start to become unreadable when you get dense areas of data. In this dataset there aren’t even that many data points (576 total), but the concentration downtown makes that area a jumble of markers.

You can tell that there are a lot of points in the area, but you can’t tell how many there are. And in this case, each marker doesn’t just represent a single point, the size of the circle also represents how many tickets were issued at that location, so ideally I would be able to look at this map and tell where the most citations are issued, but I simply have no way of knowing that.

The problem with relative sizing of individual markers

The second problem has to do with that large marker near the bottom of the map. This map leads to a confusing conclusion because every map marker, regardless of how close it is to other markers (or even if it overlaps others) is showing the value of a single location. This means that if there are 10 locations all within a single city block that each have 100 citations, and then there is a separate location elsewhere in the city that has 500 citations, that location with 500 citations will appear 5 times as large as any of the other locations, and you will have no way of knowing that within a single block there were actually 1,000 citations issued (making that block a far more likely area for receiving a parking ticket). What we really want to see is the total citations issued within a certain geographic radius, so we can view which areas have the most total citations, not just the single locations.

SpatialKey to the rescue with aggregated heatmaps

To try to better understand the underlying data, I decided to bring the same dataset into SpatialKey. The data on the SFGate website was loaded into their Google Maps application in JSON format, and to get it into SpatialKey I simply grabbed the JSON feed, opened it up in a text editor, and did a bit of find and replace to convert the data to CSV (the whole process took a few minutes to get the CSV ready for import). Then I imported the data using the SpatialKey CSV import feature (for more on this and other features, check out the feature videos).

Once I had the data imported I loaded up a new report with a map and here’s what I got:

San Francisco parking citations heatmap in SpatialKey (click to enlarge)

This map shows a heatmap that visualizes the total citations issued. But the important difference is that the clusters of data points downtown are aggregated by geography so items that are very close together are all factored into the hotspots. Now we’re able to see the real relationship between areas of the city. That point down in the southern part of the city is still visible, but it becomes clear that there are far more citations issued downtown. This deeper understanding is possible because we aren’t simply throwing a marker for each point up on the map, we’re aggregating the total value for all markers within a certain geographic area.

If we zoom in downtown we can see another view that shows the more specific hotspots:

Heatmap of parking citations in downtown San Francisco (click to enlarge)

Looks like they get people along Market street. Right at the Westfield Shopping Center is a prime spot, as well as the intersections of  Market and O’farrell and near Market and Sutter (if you’re parking there, look out!). You can see that out of the total parking citations in this dataset (82,911) about 42% (34,695) are issued just within the downtown area shown in the above screenshot.

I hope this example shows how important it is to be able to tell the right story with your data. SpatialKey gives you the flexibility to visualize your data in complex ways that go beyond simply throwing markers on a map. Have you run into similar problems with the current tools for web-based mapping? If so let us know in the comments and then sign up for the SpatialKey beta program!

The app was created as part of Intuit’s Workplace contest, and we’re proud to announce that it was chosen as the winner. This is the first non-preview release of the SpatialKey system, and we prepared a few videos to help you get started. The first shows all the steps necessary to sign up for the SpatialKey app and launch your data.

The other videos (which introduce the Data Exploration and Charting Dashboard Builder visualization templates) can be seen on our landing page for the Intuit app. For more information on SpatialKey and the Intuit Partner Platform, see these articles:

• Slick data-visualizer launched for QuickBooks users (CNet)
• Intuit Workplace Analyzes SMB (PCMag)
• SpatialKey Mashes Up QuickBooks Customer Data (Programmable Web)
• Universal Mind Wins Intuit WorkPlace Application Contest (Intuit)

On October 14, the New York Times released their Campaign Finance API, a simple interface to the Federal Election Commission’s candidate fund-raising data. Data are available for the entire country, for all primary candidates, from the beginning of the 2008 primary campaign up until the end of August. Though summary data are available, the API also allows requests for individual donors by zip code, and includes name, address, amount, and date for all donations.

With a simple script, we were able to string together requests for multiple zip codes and parse the resultant XML into a CSV file that could be loaded and geocoded with the SpatialKey Data Importer (all while staying under the Times’ generous 5000 requests/day limit). Doing so yielded complete datasets for a number of major cities and battleground states; here we present campaign donations data from the Democratic strongholds of Chicago and Denver.

Chicago donations to Obama and McCain, August 2008

Launch this visualization

This dataset includes all available donations from Chicago received by either major presidential candidate in August, the most recent month for which individual-level data are available. The over 4000 donation records provide a unique view into political spending in Obama’s home base during a month in which the candidates were nearly even in the polls.

With our new and in-development Data Exploration template, you can filter the donation records by any attribute, allowing you to exclude individual donors or candidates, or select a range of dates or donation amounts to display. Filters are immediately applied to the visualization, and many trends in the data are revealed only after such filtering has been applied. For example, displaying only donations to John McCain reveals a much different geographic trend than the unfiltered dataset, as nearly all donations from Obama’s neighborhood of Hyde Park are erased, leaving hotspots only in the downtown, Gold Coast, and far northwest neighborhoods.

Denver donations to Obama and McCain, Summer 2008

Launch this visualization

Denver provides another compelling case study of campaign fundraising, given its position as the capital of a Western battleground state, and the host of the Democratic National Convention in late August. Both candidates also visited the city multiple times throughout the summer months. The temporal trends and geographic clustering in this data can be explored via our Data Exploration template. The scented filtering widgets allow you to visualize the distribution of attributes while filtering them down to reveal previously hidden geographic trends on the map. For example, the screenshot above shows only donations over $500 to both candidates. This cuts out nearly half of Obama’s receipts, though only about one-third of McCain’s, and produces a markedly different geographic distribution than the unfiltered receipts.

Finally, though the above has concentrated on trends and aggregated visualizations, the Data Exploration template also allows you to drill down into the data to pull out individual records. After viewing the overall trend for a city or neighborhood, click anywhere on the map from which donations originated, and you can page through a list of individual donations in the vicinity, as shown in the screenshot above.

…it’s a combination map of satellite images, street map, and my geographically assigned patrol beat map in layers. The company has made it so user friendly that I can literally research and plot over 400,000 calls for service in the last 5 years in about 30 seconds. The system is web based so quickly analyzing What-Ifs can be accomplished anytime, anywhere…

Chief Greiner further highlights the ease-of-use of SpatialKey and its appeal to younger officers:

At the end of the day I want to give each of my officers the ability to do What-Ifs from home. My newest officers are gamers raised in a world of video games. I want this to become their new game of choice in helping solve crimes and arrest suspects more quickly. OPD [Ogden Police Department] officers work the same area for a year and this gives them a tool for their area to use during their un-committed time.

With SpatialKey, our goal is to put powerful visualization and reporting tools into the hands of those who need them most: the decision-makers. We continue to work with the Ogden PD and other departments to test our tools, implement their feedback, and ensure that they are able to make best use of their geotemporal data.

The article — about the use of browser-based visualizations and analytical dashboards for business intelligence — features an interview with Chief Jon Greiner of the Ogden Police Department in Utah. Ogden is the first police department in the country to implement the enterprise version of the SpatialKey Law Enforcement Dashboard:

Today, the officers are using the new BI tools to perform geographic profiling of crimes and analysis of police data “in seconds,” he says. Before, it could take days for the department’s single crime analyst to fulfill a report request. An added bonus is that experienced police officers with extensive street experience are now able to apply their firsthand knowledge to crime analysis.

It’s great to see our crime mapping tools featured in an article aimed at the broader business intelligence and IT fields, especially as we expand the focus of our tools with the SpatialKey Technology Preview and SpatialKey Personal.

The standard cartographic approach to symbolizing point data with attributes attached is the proportional symbol map.  Such maps, which are one of the three symbolization methods currently offered in SpatialKey, use symbols (typically circles) scaled proportional to point values.

Offering multiple symbolization options — each available at all times and map scales — allows users to switch to a more appropriate rendering for their dataset and switch back-and-forth to see their data in new ways.  In addition to the proportional symbols shown above, our templates currently offer two raster-based aggregated renderings: the heat grid and heat map.

In the above screenshots, the colored heat grid and heat map represent the density of Wal-Mart stores. In this case, store locations are aggregated to an arbitrary, scale-dependent grid. Thus, the brightest grid squares (and the hottest areas on the continuous heatmap representation) represent the areas of highest Wal-Mart concentration.

These renderings can also be used to show attributes of point data, in which case the hottest/brightest areas represent the areas with the highest average or total value for a given attribute. And of course mousing over the map reveals tooltips with exact values for these “quadrats” (screenshot below shows Sacramento home sales data).

The heatmap symbolization is newer, less vetted by the cartographic community, and perhaps less straightforward than the proportional symbol and heat grid renderings. But with the proper user interface, and in concert with other available renderings, we believe heatmaps can help facilitate effective visualization of geotemporal data.

We are particularly interested in feedback on the use of the heatmap symbology for mapping attributes of data (like home prices). We are excited to add additional symbolization options to our templates, as well as improvements to existing renderings, as we continue to develop the SpatialKey visualization system.

Droughts, floods, storm events — all are visible in our visualization templates utilizing this free government data.  See, for example, the Great Flood of the Missouri and Mississippi Rivers in 1993 by visualizing streamflow (in cubic feet per second) measured at dozens of stations along these rivers.

Launch this visualization

The symbols on the map are sized proportional to the streamflow readings, allowing us to see the peak of the flooding in July and August.  This dataset shows off many unique aspects of the SpatialKey visualization system.  First, the templates excel at aggregating both spatially and temporally.  In the screenshot above, the St. Louis and Grafton stations are aggregated together, because they are so close and would overlap, producing illegibility and reducing clarity, if they were displayed separately.  The average of the two stations’ readings is symbolized; zoom in to reveal more detail, separating the St. Louis and Grafton stations’ symbols:

In addition to the spatial aggregation, this example also demonstrates temporal averaging.  In the above screenshot, each of the symbols represents and average of one month of daily readings.  Selecting different time ranges on the time chart instantly reveals the new average, and any range can be animated in the Playback template.

All of the SpatialKey visualization templates include extent-filtering, which updates the time chart to only display the temporal trend for points in the current map extent.  Seeing the streamflow trend for an individual station is therefore as easy as zooming into that station:

The dataset shown off in this example is different from the others in our gallery.  While the others demonstrated an ability to aggregate and visualize thousands of independent points across space and time (home sales, Wal-Mart store openings, arrests), the dataset presented here has only dozens of points, but includes thousands of streamflow readings (the dataset is over 12,000 rows long).  This “remote sensor” style of dataset is easily accomodated by our templates.  A third type of point dataset tracks assets (shipping containers, police cruisers) whose locations and attributes (speed, weight, etc.) may change over time.  We have developed solutions in the past for this latter type of dataset, and will show off asset tracking in future visualization templates.

Please comment here or contact us if you have any comments or questions.