Did you know that you can navigate the posts by swiping left and right?

Read in Data by D3 - Part 2: D3 Request

November 16, 2017, November 26, 2017 | Comments

category: VISUALIZATION
d3 javascript

After introducing d3-dsv module from last blog, we learn how to use D3 to parse different types of flat files. To illustrate different charts D3 could achieve, a simple data set created during coding is good enough already (see this). But more often than not, it is more common to have an external data source residing in the local server or some remote server for D3’s consumption. This blog aims to introduce using d3-request module to pass different types of data files D3 could consume from the local server.

d3-request explicitly provides direct functions to be able to bring in the following types of files: html, xml, text, json, csv and tsv. We start with csv and tsv files since their structure is much simpler than others’. I used the data used in the area chart and line chart for illustration.

1. d3.csv

d3.csv(url[[, row], callback]) only requires a url to return a new request. Usually, a callback function is specified to pass the data object as a parameter to callback function once the data is loaded.

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
  d3.csv("/blog/data/browser_statistics.csv",function(data){
    console.log(data);
//(118) [{…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, …]
//[0 … 99]
//[100 … 117]
//columns:(6) ["Date", "Chrome", "IE/Edge", "Firefox", "Safari", "Opera"]
//length: 118
//__proto__: Array(0)
    console.log(data[0]);
//{Date: "Oct-17", Chrome: "76.10", IE/Edge: "4.10", Firefox: "12.10", Safari: "3.30", …}
  });
</script>

From the above example, using d3.csv converts the raw csv file into an array of objects with a key/value pair. The header of csv file’s column becomes the key and each row becomes value. And you may also notice the value is actually a string even though it should be numbers since they are the market share percentage in this example. This may not be what you actually look for. There are a few ways to address this. First, we can use the first optional parameter of d3.csv function row to map and filter row objectss to a more-specific representation,

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.csv("/blog/data/browser_statistics.csv",function(data){  
    return{
        Date: data.Date,
        Chrome: +data.Chrome,
        "IE": +data["IE/Edge"],
        Firefox: +data.Firefox,
        Safari: +data.Safari,
        Opera: +data.Opera
        };    
  },function(data) {
  console.log(data[0]);
  });
</script>
//{Date: "Oct-17", Chrome: 76.1, IE: 4.1, Firefox: 12.1, Safari: 3.3, Opera: 1.2}

which is also equivalent to:

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.csv("/blog/data/browser_statistics.csv")
       .row(function(data){  
    return{
        Date: data.Date,
        Chrome: +data.Chrome,
        "IE": +data["IE/Edge"],
        Firefox: +data.Firefox,
        Safari: +data.Safari,
        Opera: +data.Opera
        };    
     })
   .get(function(data) {
  console.log(data[0]);
  });
</script>  
//{Date: "Oct-17", Chrome: 76.1, IE: 4.1, Firefox: 12.1, Safari: 3.3, Opera: 1.2}  

In this way, you also have the full control over how the data is processed. For example, I modified the original “IE/Edge” with “IE”.

We can also just use the callback function alone and use forEach to iterate over the whole array objects and combine it with + to convert the original strings into numbers,

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.csv("/blog/data/browser_statistics.csv",function(data){  
    data.forEach(function(d) {
    d.Chrome = +d.Chrome;
    d["IE/Edge"] = +d["IE/Edge"];
    d.Firefox = +d.Firefox;
    d.Safari = +d.Safari;
    d.Opera = +d.Opera;
  });
  console.log(data[0]);
  });
</script>  
//{Date: "Oct-17", Chrome: 76.1, IE/Edge: 4.1, Firefox: 12.1, Safari: 3.3, Opera: 1.2}

which is equivalent to:

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.csv("/blog/data/browser_statistics.csv")
       .get(function(data){  
    data.forEach(function(d) {
    d.Chrome = +d.Chrome;
    d["IE/Edge"] = +d["IE/Edge"];
    d.Firefox = +d.Firefox;
    d.Safari = +d.Safari;
    d.Opera = +d.Opera;
  });
  console.log(data[0]);
  });
</script>  
//{Date: "Oct-17", Chrome: 76.1, IE/Edge: 4.1, Firefox: 12.1, Safari: 3.3, Opera: 1.2}

2. d3.tsv

tsv file is the same as csv file except the delimiter is tab instead of comma. We can use the same method as d3.csv by using d3.tsv function instead. There is no need to repeat here.

However, if the delimiter is something other than comma or tab, we can rely on d3-dsv module to create a formatter first. And use d3-request module to call d3.request, mimeType and response functions to pass the other delimited files to D3 as the example below shows:

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
  var ssv = d3.dsvFormat(";");
  d3.request("/blog/data/age_by_gender.txt")
    .mimeType("text/plain")
    .response(function(xhr) { return ssv.parse(xhr.responseText) })
    .get(function(data) {
    console.log(data[0]);
  }); 
</script>  
//{Age_Group: "5 to 9 years", Male: "0.066", Female: "0.062"}

As expected, the semi-colon delimited file is converted into an array of objects and all properties are strings. If this is not something we need, we can either pass row parameter into response function to convert properties into numbers:

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
  var ssv = d3.dsvFormat(";");
  d3.request("/blog/data/age_by_gender.txt")
    .mimeType("text/plain")
    .response(function(xhr) { return ssv.parse(xhr.responseText, row) })
    .get(function(data) {
    console.log(data[0]);
  }); 
function row(d) {
  return {
    Age_Group: d.Age_Group,
    Male: +d.Male,
    Female: +d.Female
  };
}
</script>  
//{Age_Group: "Under 5 years", Male: 0.064, Female: 0.059}

Or use forEach to pass into callback function and iterate each object of array to convert into numbers:

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
  var ssv = d3.dsvFormat(";");
  d3.request("/blog/data/age_by_gender.txt")
    .mimeType("text/plain")
    .response(function(xhr) { return ssv.parse(xhr.responseText) })
    .get(function(data) {
      data.forEach(function(d) {
        d.Male = +d.Male;
        d.Female = +d.Female;
    });
    console.log(data[0]);
  }); 
</script>  
//{Age_Group: "Under 5 years", Male: 0.064, Female: 0.059}

3. d3.text

d3.text only has two parameters, url and callback function. It is essentially the same as the example illustrated in d3.tsv section with default mime type “text/plain”. Nothing new.

4. d3.json

Like d3.text function, d3.json also has two parameters, url and callback function. It only differs in the mime type with default value “application/json”. I use the json data set used in the cluster diagram as well as tree diagram as example.

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.json("/blog/data/all_countries_2015.json",function(data){
    console.log(data);
  });
</script>  
//{data: {…}, children: Array(6)}

Again, we can see using d3.json would convert the json file into an array of object and we can manipulate the array with the same method introduced in the section above.

5. d3.html and d3.xml

Reading html and xml file is pretty much the same as reading json or text file. The key difference is that html has default mime type “text/html” and xml has default mime type “application/xml”. The syntax and usage is quite similar to d3.text or d3.json. I created a simple .xml file for illustration purposes.

<script src="https://d3js.org/d3-collection.v1.min.js"></script>
<script src="https://d3js.org/d3-dispatch.v1.min.js"></script>
<script src="https://d3js.org/d3-dsv.v1.min.js"></script>
<script src="https://d3js.org/d3-request.v1.min.js"></script>
<script>
   d3.xml("/blog/data/example.xml",function(data){
    console.log(data);
    console.log(data.documentElement.getElementsByTagName("body"));
  });
</script>  

d3-request provides a full stack of parsing different types of data format for its consumption. Through different built-in functions introduced above, we could also generalize the commons between each other.

In general, the six specific types of D3 function are just transformation of generic d3.request().mimeType().response().get() calls which applies to different types of format. There are two kinds of plain files: text(csv, tsv, text & html) and application (json & xml). We just need to specify those types in mimeType() function and pass it into response() function. Here is a summary:

////Read csv file
//Direct call:
d3.csv(url[[, row], callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("text/csv")
  .response(function(xhr) { return d3.csvParse(xhr.responseText, row); })
  .get(callback);
  
function row(d) {
  return {
  //data manipulation here...
  };
}  

////Read tsv file
//Direct call:
d3.tsv(url[[, row], callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("text/tab-separated-values")
  .response(function(xhr) { return d3.tsvParse(xhr.responseText, row); })
  .get(callback);

function row(d) {
  return {
  //data manipulation here...
  };
}  

////Read other delimited-separated file
var dlm = d3.dsvFormat("delimited"); //create formatter with specific delimiter
d3.request(url)
  .mimeType("text/plain")
  .response(function(xhr) { return d3.dlmParse(xhr.responseText, row); })
  .get(callback);

function row(d) {
  return {
  //data manipulation here...
  };
}  

////Read text file  
//Direct call:
d3.text(url[, callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("text/plain")
  .response(function(xhr) { return xhr.responseText; })
  .get(callback);
    
////Read json file  
//Direct call:
d3.json(url[, callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("application/json")
  .response(function(xhr) { return JSON.parse(xhr.responseText); })
  .get(callback);
    
////Read html file
//Direct call:
d3.html(url[, callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("text/html")
  .response(function(xhr) { return document.createRange().createContextualFragment(xhr.responseText); })
  .get(callback);  
  
////Read xml file
//Direct call:
d3.xml(url[, callback]);
//which is equivalent to:
d3.request(url)
  .mimeType("application/xml")
  .response(function(xhr) { return xhr.responseXML; })
  .get(callback);


Reference:

(1). D3 API Reference, https://github.com/d3/d3/blob/master/API.md.
(2). d3-request API, https://github.com/d3/d3-request.
(3). Data Loading in D3, http://www.tutorialsteacher.com/d3js/loading-data-from-file-in-d3js.
(4). Learn JS Data, http://learnjsdata.com/read_data.html.