Hackpads are smart collaborative documents. .

Jen Lowe

1478 days ago
Unfiled. Edited by Jen Lowe 1478 days ago
Jen L Data Questions and Answers
 
Hi all - I'm working through people's questions in order of what I can answer quickest. Please add more questions or info as you want.
 
motoishmz Motoi's location data
My OpenPaths data (latest 100 records):
 
If you want to plot locations on a map, http://geojson.io/ is AMAZING. LOOK:
 
 
 Steps to make this map in ~40 sec:
 
 1. Copy / Paste Motoi's location data into an OpenOffice spreadsheet
 2. Save the spreadsheet as a csv
 3. Drag the csv file onto the right hand side of http://geojson.io/
 4. Click save
 5. Share the gist link!
 
 ***Next steps here: heatmap! Something actually interesting!
 
 -----------------------------------------------------------
 
 THE STORY OF JSON
 
 Collecting good json tools
 
 json is a data format, like csv (comma separated values) or tsv (tab separated values)
 
 json has a lot of curly braces {} and brackets [] - when you see these in a data file, you're probably working with json
 
 json stands for "Javascript Object Notation" OBJECT is the most important part of that - json is good at describing OBJECTS. That's why data people like json.
 
 For instance! If I made a csv table of the data you're interested in it would look like this:
 
  • name, data, format, link
  • Ishac, google searches, , 
  • Ishac, browser history, , 
  • Jason, OpenNI from a dancer, , 
  • Jason, Audio from a singer, , 
 
If I use the AMAZING Mr. Data Converter http://shancarter.github.io/mr-data-converter/ to convert that to json, it looks like this:
 
  • {"name":"Ishac"," data":" google searches"," format":" "," link":" "},
  • {"name":"Ishac"," data":" browser history"," format":" "," link":" "},
  • {"name":"Jason"," data":" OpenNI from a dancer"," format":" "," link":" "},
  • {"name":"Jason"," data":" Audio from a singer"," format":" "," link":" "},
 
See? {} and [] everywhere. That's json.
 
If I put this json into the http://jsonlint.com/ json validator and click "Validate" it formats it nicely, and tells me I have valid json. Now you can see better how json works - each line of a csv becomes an object in json. Every single piece of data comes with a name first, basically what would be the column name if the data were in a csv. (Scroll down and I'll tell you why we sometimes use json and not csv.)
 
[
    {
        "name": "Motoi",
        " data": " location",
        " format": " csv",
        " link": " http://motoishmz-location.herokuapp.com/"
    },
    {
        "name": "Motoi",
        " data": " location",
        " format": " json",
    },
    {
        "name": "Jonathan",
        " data": " Met",
        " format": " ",
        " link": " http://scrapi.org/"
    },
    {
        "name": "Ishac",
        " data": " google searches",
        " format": " ",
        " link": " "
    },
    {
        "name": "Ishac",
        " data": " browser history",
        " format": " ",
        " link": " "
    },
    {
        "name": "Jason",
        " data": " OpenNI from a dancer",
        " format": " ",
        " link": " "
...
1463 days ago
Unfiled. Edited by Casey Gollan , Jen Lowe , Ishac Bertran , Zach Lieberman 1463 days ago
Casey G Postcard Spots
 
We have 250 postcards for the show to scatter across the city. Where should they go?
 
If there's a place you're going today or tomorrow, you can drop em off, otherwise just drop places on the list and I will just bike around. 
 
If you wanna grab cards for your friends, I'll leave them on the table next to the green bins.
Jen L
  • thanks! Can you save a couple for me?
 
Casey G
  • Eyebeam (duh)
Zach L
  • 33 flatbush / elevator
Casey G
  • Parsons
  • ITP
Jen L
  • Printed Matter (art book bookstore / 195 Tenth Avenue btwn 21st & 22nd)
  • 192 Books (across from Printed Matter- they're friendly & have lots of art books)
  • Taeyoon Choi's favorite gallery - I know he mentioned it once, but don't remember the name
Ishac B
  • SVA (going today, will drop some)
 
 
1467 days ago
1472 days ago
Unfiled. Edited by Mini Kim , Jason Levine , Jen Lowe , Rachel , Tega Brain 1472 days ago
7:00 - 7:15 Mini
 
Jen L
  • anyone around who can help with this? I looked at the process and just don't know - I can OAuth twitter, but can't quite figure out OAuth hackpad. Thx!
Rachel
  • ok, just sent out an email-- hopefully someone can help clarify:) 
 
 
8:15 - 8:30 TEGA
 
 
 
8:45 - 9:00 Jason 
 
1475 days ago
Unfiled. Edited by Tega Brain , Jen Lowe , Simona De Rosa , Zach Lieberman 1475 days ago
Tega B DataVis with Jen
 
When looking for data behind data visualisations. 
Use google developer tools and open the html of the site.
  • In the html we are looking for links to the json files or equivalent to find the data used in the visualisations. 
  • Looking at the peoplemovin website. 
  • Look for json but if there is none.
  • look at javascript files. 
  • In the javascript there is a php script to connect to some json files in this example.
  • This takes us to the data.
 
JSON TOOLS
JSON Validators are really useful for checking JSON files. 
 
 
Binning 
  • Taking a dataset and dividing it up into bins (or sections). It means a person has selected where the important distinctions in the data. It is a way that data cab be manipulated (for good or evil). 
 
  • How does data visualisation obscure the assumptions upon which it is based?
 
Working with APIs
  • Some are well documented, some are not. 
  • Looking at forecast.io
  • An API call - this is an call that will return a slice of the data to you, you can type this into your browser and the data will be returned to you. 
  • You are requesting just a slice of the data from the database. 
 
Python is great for working with data- it has a good community and history. 
  • Python wrapper libraries have been written for many APIs. 
  • The forecast API, has a few python wrappers
  • PIP is a thing that takes care of the installation for you. 
  • If you can avoid downloading zips of python and installing AVOID THIS!
  • PIP is a thing that will manage the installation of the python stuff for you. Saves you a lot of complications of the manual installation. 
 
Using Python with Homebrew.
5 STEPS FOR WORKING WITH PYTHON WITHOUT LOSING YOUR MIND
  • go to the homebrew webpage.
Jen L
  • Install homebrew
  • Install python: $brew install python
  • *change the path for python - you may or may not need to do this - use if pip isn't found* 
  • $nano .bash_profile
  • add this line to your .bash_profile: export PATH=/bin:/usr/local/bin:$PATH
  • save your .bash_profile
  • Now you can install using "pip install" to install python libraries
 
pip install requests
pip install python-forecastio
 
Sublime Text + Python
 
Save this as test.py in SublimeText:
 
 
import requests
import forecastio
 
print 'hello python!'
 
api_key = "31f38d07c51ba765afceb5189f2906e5"
lat = -31.967819
lng = 115.87718
 
forecast = forecastio.load_forecast(api_key, lat, lng)
 
byHour = forecast.hourly()
print byHour.summary
 
Hit Command-B - it should run......
 
 
Tega B TERMINAL STUFF
  • if you have a giant file that you cannot open, you can use the terminal to type head and the file name and it will just open the file header. 
Zach L
  • likewise "tail" which opens the end
 
 
 
 
 
 
1475 days ago
Unfiled. Edited by Jen Lowe 1475 days ago
I'd like to know how to use OAuth! how to set up the files and request the keys, etc.
 
Jen L Simona: Find the structure of a json using Processing
 
how to access hackpad api
 
 
 
1479 days ago
Unfiled. Edited by motoishmz , Jen Lowe , Casey Gollan , Jonathan Dahan 1479 days ago
Metropolitan Museum artwork + metadata: http://scrAPI.org
 
The searches using Google during the last weeks. Or the browser history, and how links are connected (branches, from searches, etc.)
Jen L
  • Can someone get me a link / download / something of an example of the data you've been collecting on searches? Jonathan Dahan?? Have you been collecting this? Or Ishac Bertran do you mean your own Google searches? Thanks!!
Jonathan D
 
OpenNI data from a dancer
Audio Data from a singer
Jen L
  • Jason Levine can you share an example of what this data looks like?? Or even just describe what values you can get out from these dancers / singers? Thanks!
Casey G
  • Jen Lowe if you type "//" at the beginning of a line it'll format it like a comment and you can indent to have a threaded conversation.
Jen L
 
 

Contact Support



Please check out our How-to Guide and FAQ first to see if your question is already answered! :)

If you have a feature request, please add it to this pad. Thanks!


Log in