CDE's SchoolVIEW

SchoolVIEW data in SpatialKey

I was interested in seeing the correlation between these different metrics, and (since we’re obsessed with location) how that correlation relates to geography. So, I imported that data into SpatialKey.  The source file was a CSV with a row for each school.  Here’s what that data looks like:

SpatialKey’s bivariate renderer allowed me to quickly explore the data in just that manner. The bivariate renderer allows you to select two numeric attributes in your dataset, and an aggregate calculation for each. In the image below, I selected average math growth percentile and average math proficiency.  Each dot in the scatterplot legend at the upper right represents a colored location (grid cell) on the map.  The position of the dot represents its relative score for average math growth (y axis) and average math proficiency (x axis).  The color “behind” the each dot is the color used for the corresponding grid cell on the map.

This visualization shows the coorelation between math proficiency and growth, as it relates to location. (Click the image for a larger view.)

We can see there is a general positive correlation, where most locations have a similar relative score for math performance and growth: Most points on the scatterplot are along an imaginary diagonal line from the bottom left (low in both metrics) to upper right (high in both metrics).  What’s often interesting and informative is to see areas that deviate from the norm in terms of the correlation.  Areas with relatively high proficiency but low growth – “strong but losing ground” – are colored blue, while areas with low proficiency but high growth – “risin’ up” – are colored red.  These are both negative correlations.  Areas that score low on both metrics are shaded white, while those high in both attributes are shaded black – both positive correlations.  For this type of visual analysis, areas that fall toward the middle of both ranges are usually less interesting, and so those colors are more transparent to allow you to focus on the extremes.  It may take a few seconds to orient yourself to this view, but once acclimated it’s a powerful way to visualize some complex – and otherwise difficult to express – relationships.

You can correlate any pair of attributes by simply selecting from one of the axes in the scatterplot legend. This next image compares average math and reading proficiency.  First, notice there seems to be an even stronger correlation between these two variables than the previous set.  (The points line up even closer on the imaginary diagonal line.)   It’s also interesting to compare these two images; Notice how the schools in some locations are relatively strong (shaded black) or weak (shaded white) in both visualizations, while others show a particular weakness in one of the metrics.

Selecting a point on the scatterplot shows the corresponding location on the map. In this case, we've highlighted a school that is an outlier because it's relatively strong in math versus its perforrmance in reading, realtive to other schools. We can easily see this school is in Moffat County. (Click the image for a larger view.)

SpatialKey makes it easy to uncover and visualize these relationships, and to share them with others. From uploading the spreadsheet with school data to presentation, this only took a few minutes to create – without any programming or hassle. And, this is just the start. By adding filters we can see these trends for schools of certain sizes or types, or compare these trends over time.

Further Analysis

An interesting next step would be to see if there is any correlation between the areas that deviate from the norm school performance and property value changes. For example, are the “rising up” areas ones where real estate values have been growing faster than average, or gentrification is taking place. (Of course, determining causality is a whole different conversation!) One could bring additional real estate or demographic data into SpatialKey to help answer those questions. SpatialKey makes it easier to understand the relationships between disparate datasets.

Try it out for yourself

Don’t take our word for it. You can start uploading your own data and visually correlating it right away by signing up for the 30-day trial of SpatialKey. Or, contact us and we’ll be happy to walk you through the process.

We’ll compare a zip-code thematic map with a heatmap. Both maps show average home sale price by geographic area (either zip codes or clusters of points). The image below shows the two map types side by side.

Now we’ll step through an analysis of these different map types to see why they produce different views of the same data.

Thematic map by zip code

First, let’s take a look at mapping home sales in Sacramento by zip code. The map below shows thematic zip codes colored by the average sale price. You can see the highest range is $400,000 and up and includes 3 zip codes in the image below. I want to focus on comparing the two labeled zip codes, 95818 and 95822. You can see that the 95822 zip code area has a much lower average sale price than 95818, which is immediately north of it.

Density heatmap with zip-code boundaries

However, if we switch to a density heatmap we see a different picture. Switching from thematic zip codes to a density map takes literally 3 clicks in SpatialKey. The map below shows average sale price as a density map, with the boundaries of the zip codes overlaid in red. This is the exact same data showing the exact same attribute (home sales showing average sale price). But if you compare this image with the thematic map above you’ll notice that the hotspots tell a different story. A fluid area that overlaps both the zip codes we looked at above is actually the area with the high average prices. That area doesn’t cleanly fall into a single zip code.

This isn’t too shocking, since it intuitively makes sense that fairly arbitrary boundaries like zip codes wouldn’t directly map to more or less expensive areas of town. But it illustrates the difficulty of rendering your data thematically by certain shapes, like zip codes or counties.

Density heatmap with neighborhood boundaries

To further analyze the dataset I decided to load in the boundaries of the neighborhoods in Sacramento (the file was downloaded here). Now we see boundaries that come much closer to matching the home prices. Intuitively this also makes sense; if you think about home prices in your city you’ll likely think of expensive and cheap neighborhoods, not zip codes.

Everything has its place

Both thematic maps and density maps are useful when exploring geographic data. Both show you important aspects of your data, but it’s important to keep in mind the inherent limitations of the different methods. With SpatialKey, we provide you with the tools to easily switch back and forth between these rendering methods in seconds.

Try it out for yourself

You can start uploading your own data and making thematic maps right away by signing up for the 30-day trial of SpatialKey.

Photo courtesy of CNN and Google maps.

On the business side, 80% of business data has a location component which provides a goldmine of untapped information for marketing, sales and operations. But current visual mapping and analysis tools are expensive, can only be accessed by trained specialists, and require heavy IT involvement to set up and maintain. This is a big barrier to entry for most businesses. They want to “see”, understand and communicate data trends, but don’t have the time nor means to invest in yet another expensive infrastructure.

The businesses that already do leverage visual mapping and analysis can more effectively and more quickly see geographic or time-based data and trends critical to sales and operations. This provides them a real competitive advantage. Many oil and gas companies for example have invested in sophisticated Geographic Information Systems (GIS) and brought in GIS specialists to gain insight on their location intelligence via visual maps. This allows them not only to plot areas with the highest potential to drill in, but also better manage their pipelines, operations, retail facilities, and more.

…. or….    . …and…            … and..     

Thankfully, a revolution is taking place that allows “regular” business users -with no GIS training nor deep pockets-  to leverage the power of visual mapping and analysis. Enter Software as a Service (SaaS). SaaS is transforming mapping and data visualization in the business world the same way Google Maps revolutionized mapping for consumers. Using cost-effective, user friendly SaaS mapping and analysis applications, such as SpatialKey, organizations of all types and sizes can now import their business data, combine it with geographic or competitive information, and start visually analyzing trends critical to their business. Where are key customers located? How can they maximize results in their sales territories? How best to map their sales territories? Where should they open a new retail outlet? How does Q2 sales compare to Q1 on a geographic basis? What marketing campaign resulted in the highest ROI? And so much more.

Opportunities and threats previously hidden within row and column-based datasets are now clearly visible via interactive maps. Concepts difficult to explain in text or PowerPoint presentations can now also be shown and therefore easily understood resulting in better decision making. What’s more, since everyday decision makers can use these applications, “what if” questions can be answered on the fly versus having to wait for an analyst to do a new data query. Decision-making, communication, and collaboration are improved. After all, seeing is understanding and believing, even in the business world.

Note: we’ll be adding blog posts around visual mapping for sales and marketing users over the next few weeks. In the meantime you can find out more at our sales and marketing and/or enterprise solutions pages.

Find some data

Your data must have location details down to the level of granularity that you are trying to map. For instance, if you want to show a map of states, all the records in your data should at least have US state (your data can be more granular too, you can map address-level data by state if you want). In this example I’ll be mapping US unemployment rate. The data for unemployment is provided by the Bureau of Labor Statistics and can be found here. I took the latest stats by US county and extracted only the data for October 2009.

After just a little massaging in a spreadsheet program my data looked like this:

You can download the CSV file that I used if you’d like to try it out for yourself.

Upload to SpatialKey

Once you have your data ready, you can upload it to SpatialKey. If you don’t already have a SpatialKey account, you can sign up for the free 30-day trial to get access right away and start uploading. During the upload process you’ll be asked to identify the location columns in your data, like street address, zip code, etc. We’ll do our best to automatically identify these columns based on your data, but you might have to help us out.

Make your map

When you load your data onto a map you’ll be asked what kind of map you want to create. We’ll make a thematic shape map, and we’ll choose to map the data by US geography (this includes state, county, or zip code).

Then we choose what the map should display. In this example we want to show the unemployment rate, so I’ll pick average unemployment rate, which will start me off with  a map of the US states with the average unemployment rate for all the counties in each state.

Now our thematic map shows the average unemployment rate for all the counties aggregated by state.

I can switch between this state view of the data and counties by changing the options in the layer’s settings panel.

Now we have a map of all the counties in the US (including Puerto Rico) that shows the unemployment rate of each county.

Customize and Explore

You can easily customize the bin ranges if you want to tweak them, or you can control the colors used (all maps are the same, just with a different color scheme):

You can also use all the filtering options that SpatialKey offers to filter the data in your thematic maps. Here’s an example of filtering to only show counties where the total labor force is over 100,000.

And here’s another example to show only the counties where the unemployment rate is greater than 15%:

No Programming Required

To generate these maps you don’t have to write a single line of code. It’s as simple as uploading your data and stepping through a few guided steps. If you wanted to change the map to show the total labor force per county instead of the unemployment rate, it only takes 3 clicks. There are lots of ways to make these maps, like this great tutorial on FlowingData shows, but we think SpatialKey gives you the easiest way to create and analyze thematic maps.

We mapped the concentration of where these permits were issued. The different types of buildings, such as apartments versus office buildings, have very different distributions throughout the city. Some of these distributions are expected, such as the high concentration of permits for offices in the downtown area of San Francsico. But some of the distributions are more interesting and tell a story about the urban makeup of the city. Notice that apartments are much more concentrated closer to downtown in neighborhoods like the Tenderloin, Nob Hill, and Hayes Valley, most heavily around the eastern and northern areas surrounding the financial district. Two-family homes (ie duplexes) have a different concentration that includes neighborhoods like Cow Hollow and the Mission. And one-family homes are in neighborhoods like Pacific Heights, Noe Valley, and Twin Peaks.

The maps here show the number of building permits by the type of building.

Here are some alternate screenshots that are at a more granular resolution, so you can see a bit more detail on the different areas of the city. Click each thumbnail for a much larger version:

Try it for yourself

You can open up the sample report that we created to visualize these building permits.  The report will load with two layers: the building permits and the neighborhood boundaries of San Francisco. You can change which types of building permits are shown by selecting items in the “Proposed Use” filter pod that is open in the report.

• Try Sample Report

Looking for easy-to-use location intelligence from your own data?  Get started with our free trial, and start visualizing your data in minutes without installing any software.

Notes on the data

It’s always important to remember what data we’re looking at. This is the number of building permits issued between September 2008 and August 2009. A single building might have multiple permits issued, which could be everything from renovations or re-roofing to a change from residential to commercial, etc.

This is the third part of an ongoing exploration of publicly accessible San Francisco data from DataSF.org. Please see the other posts in the series.

The images and reports in this post were created with publicly accessible data. We have no association with the city of San Francisco (but we’d love to, so contact us if you’re from the San Francisco government and want to use SpatialKey).

The first few rows of the Sacramento dispatch export

It took just a couple of minutes to import one of these files into SpatialKey, where we can produce rich interactive maps and reports related to dispatch activity in Sacramento.  The file from August 2009 contained about 30,600 records with location information.  Dispatches for Disturbance-Noise were the 7th most common type of dispatch in August.

Heatmap of the 1079 Noise Disturbances in Sacramento during August 2009.

By using the Temporal Heat Index and Timeline to inspect the date and time of occurrence, we get a better picture of when Disturbance-Noise calls occurred.  The Temporal Heat Index summarizes the number of crimes by hour of day and day of week.  Notice the dispatch volume is generally highest late at night – especially on Saturday and Sunday.

Most dispatches related to noise happen late at night on weekends.

So where are these early morning disturbances?  Simply select day/hour grids of interest and zoom in to see the detail.  Here’s a look at noise disturbances southeast of Capitol Park between 1 and 3am on weekends in August:

Southeast of Capital Park might be a good place to party but a hard place to sleep on weekends.

Try it for yourself

You can open up the sample report that we created to visualize these dispatches.  The report is fully interactive, so you can really explore the dispatch activity in Sacramento.  We saved the report with a filter for Disturbance-Noise. Try modifying this filter – and adding others – to see how SpatialKey works.  Interested in seeing where and when the 338 Drunk Suspects were encountered?  Or the 27 Shooting into Inhabited Dwellings?  Maybe you want to see where and when the 2246 Subject Stops occurred.  It’s simple with SpatialKey.

• Try Sample Report

Looking for easy-to-use location intelligence from your own data?  Get started with our free trial, and start visualizing your data in minutes without installing any software.

Notes

The images and reports in this post were created with publicly accessible data.  Check out Sacramento’s dispatch page to see their notes about what data is included.  We omitted data without address location in our SpatialKey report.

We have no association with the Sacramento police department (but we’d love to, so contact us if you’re from the Sacramento PD and want to use SpatialKey). SpatialKey does have specific features designed for Law Enforcement.

With the abundance of this raw data new challenges arise. How do cities display the information in meaningful ways without complex and costly software?  This data is typically shared in the format of CSV (comma separated value ), spreadsheet or XML files containing many thousands of rows. Extracting meaning from the data can be a daunting task requiring multiple pivot tables, graphs, and filters along with the expertise in doing so.  And assuming you are able to get this far, you will still be left without an easy solution in which to share this data with others.

How do you expose the location based information within the data?  It is possible take a handful of the items in the spreadsheet and plot them in a web based mapping solution but most web based maps fall short in their ability to plot thousands of points in a meaningful way.

With SpatialKey you can take nearly any of these data feeds and transform them into an interactive report in minutes. You don’t need to be a specialist to create and share time- and location-based analyses.

To demonstrate the power and flexibility of SpatialKey an example from the New York Department of Sanitation containing graffiti locations is shown below.  You can find this data in the NYC Data Mine by searching for the keyword “graffiti”.  This dataset contains requests to clean graffiti (other than bridges or highways) received from the public in the last 12 months. They include location information, open and closed dates, and details about the community.  A small snapshot of this data is shown below

Figure 1.0 - Sample of graffiti data from the NYC dataset

It is important to note that SpatialKey was not developed for this specific data and no programming was required to build the reports. It is as simple as exporting a CSV from excel and importing the data into SpatialKey. SpatialKey inspects the data during the import process and detects the data types (text, numbers and dates) and builds a custom user interface tailored to the data structure from the spreadsheet. SpatialKey also handles the geocoding as long as you have address information or X/Y in the data. The import process can be performed with thousands of rows in just a few minutes.

After importing the data a full screen map report is opened as shown in Figure 2.0. The report contains a timeline that highlights the trends of open graffiti reports over the last twelve months and the map highlights hot spots for reported graffiti locations. You can instantly identify these trends and hotspots quickly in SpatialKey then start to drill down to identify additional trends with the filtering tools.

Figure 2.0 - A map and timeline of reported graffiti

In Figure 2.1 a categorical pod for the “status” field in our data is opened and the data is aggregated by the unique statuses in that field.   By clicking on the “Closed” status you can filter out all closed incidents and the map reflects only open and pending incidents. In addition you can switch the timeline to show unfiltered data to see the trend of open/pending vs closed incidents.  Within the stacked bar chart the open incidents are displayed with the filled area and the closed incidents are shown in the unfilled area.

Figure 2.1 - Displaying the trend of open versus closed reports over time

A custom interface to display and filter data can be built in seconds with no programming or development needed. In Figure 3.0 four categorical pods have been added from different fields available in the graffiti dataset these pods can be used for both display and filtering.

Figure 3.0 - Adding pods for several fields from the graffiti dataset, pods can be used for display and filtering

Here are a few other  examples highlighting the power when you combine city data with SpatialKey.

• Crime in San Francisco’s Urban Renewal Area
• Visualizing SFPD’s “Operation Safe Schools”

Spreadsheets come to life and provide new meaning with just a few simple steps in SpatialKey. Try it out yourself with our 30 day trial.

While at ICCM2009, we’ll be discussing some of our work with ACLED (Armed Conflict Location and Events Dataset).  The ACLED project was founded by Dr. Clionadh Raleigh and team members from the PRIO Centre for the Study of Civil War, who set out to create a conflict database that would help answer researchers’ questions.  ACLED was subsequently funded by the World Bank with the aim of better understanding events in client states.

This video gives a brief overview of how SpatialKey is being used to investigate the relationship between this data and the location of refugee camps and environmental factors. What’s significant is that SpatialKey is making it easier for ACLED researchers to collaboratively bring together vast amounts of information from a wide variety of sources in a meaningful way – all without having to involve programmers or GIS experts to get their work done.

A recent Reuters article about trends in crisis mapping highlights the very issue SpatialKey is helping researchers solve:

Researchers have used maps to visualise crises for many years.

But there are drawbacks in the the use of highly-sophisticated, computerised Geographical Information Systems (GIS), which are usually used in such work — not least that they are expensive and difficult to operate.

Nor do these systems allow for much integration and collaboration, and due to their complexities they are not usually updated in real time.

For more details on our work with ACLED, read the ACLED crisis mapping case study.

It’s amazing to see how concentrated this data is around the Tenderloin and a few other areas (along Mission St near 16th also has a high concentration). Take a moment and look at the image above (click to get a larger view). If you were to draw a line around the worst parts of San Francisco what would that shape look like? Maybe something like this?

What’s amazing is that this shape was drawn by the city when they defined an “Urban Renewal Area.” The shape is available to download from the city’s GIS catalog. That shape was defined in 2002 as part of a federal Housing and Urban Development (HUD) program. One of the key goals in targeting that areas was to reduce crime. The following paragraph defines one of the five stated goals when the area was defined:

Reducing Crime
The City of San Francisco has a crime reduction strategy that is a multi-layered approach addressing: general crime prevention education; programs targeting at-risk youth (prevention); diversion programs; programs targeting incarcerated individuals to reduce recidivism; and post-incarceration programs to reduce recidivism.

That was almost 8 years ago, but the crime data in the image above is as recent as last week. There are of course many ways to interpret this data. It could indicate that the city is not making much progress reducing drug crimes within the Urban Renewal Area. Or it could mean that the city is actively targeting the area (which will lead to higher arrests) and that the indication of higher concentrations in the Urban Development Area is a good thing.

Comparing the Urban Renewal Area to the rest of SF

To put things in perspective we can compare the crime within the Urban Renewal Area with all the crime outside of the area. The image below shows the map filtered to include only crimes within the Urban Renewal Area. The report includes the total count of crimes within that area, as well as the total count of crimes in the city as a whole (including the area).

We can see that in the entire city there were 14,653 crimes. Out of those crimes, 4,291 occurred in the Urban Renewal Area.

29% of all crime in SF occurs in the Urban Renewal Area

The above statistic is for all types of crime (of which theft is the most common in the city overall). We can see by the distribution by crime type that there’s a difference in the ranking between the city as a whole and the Urban Renewal Area. Inside the Urban Renewal Area, narcotics crimes are the number one crime type (versus theft in SF as a whole). The report is even more interesting when we filter to only look at drug crimes:

Out of 2,673 total drug crimes, 1,875 occur in the Urban Renewal Area.

70% of all narcotics crimes occur in the Urban Renewal Area

Again, please keep in mind that the data only shows where the police are arresting people (see notes section below). And this is only one slice of time, from July 25 to September 20, so we don’t have the previous historical data to see any kinds of long term historical trends.

Try it for yourself

You can open up the sample report that we created to visualize these crimes. The report loads with the crime data on the map filtered to only include drug crimes within the Urban Renewal Area. It also includes another instance of the same crime dataset loaded in (but not shown on the map). That second dataset is used to show the total count of all crimes in San Francisco as well as the distribution by type of crime for the entire dataset.

• Try Sample Report

Notes on the data

The police department provides a feed of the last 90 days of crime incidents as a shapefile. The dataset used in the article was downloaded on September 21, 2009 and includes crime records from June 25, 2009 to September 20, 2009. Additionally, the city’s GIS department provides the shapefile for the Urban Renewal Area.

It is important to note that this is data of active police activity (ie arrests, citations). This means it is not a comprehensive dataset of where crime occurs. This is data about where the SFPD is arresting people. This is an important distinction to keep in mind. An increase in the number of arrests in a certain area may very well be a very good thing indicating that the police are focusing on certain problematic areas, not necessarily that actual crime in those areas is on the rise.

This is the second part of an ongoing exploration of publicly accessible San Francisco data from DataSF.org. Please see the other posts in the series.

The images and reports in this post were created with publicly accessible data. We have no association with the San Francisco police department (but we’d love to, so contact us if you’re from the SFPD and want to use SpatialKey).