Example #22 - Receiving data via paging

For a couple of the API requests that return large amounts of data, you can receive the data in pages.  Instead of waiting for and then receiving one very long stream of output, you can ask for and receive consecutive chunks - pages.

Here are two examples:  receiving the metadata for all views at a site and receiving all rows in a view.

Paging the return of all views
This uses the /api/views API service with two parameters.  page is the number of the page to retrieve.   limit is the number of views to return; the size of the page. The code prints out the names of all the views currently available on the Cook County site.  Note the first page number is 1, not 0.

Code

from urllib2 import urlopen

from json import load

hostname = "datacatalog.cookcountyil.gov"

pageSize = '20'

pageNum = 1

while True:

    url = "http://%s/api/views.json?page=%s&limit=%s" % (hostname, pageNum, pageSize)

    response = urlopen(url)

    views = load(response)

    numViewsRcvd = len(views)

    if numViewsRcvd == 0:

        break

    print '\n======  PAGE %s - %s VIEWS DOWNLOADED:  ======' % (pageNum, numViewsRcvd)

    for view in views:

        print view['name']

    pageNum += 1

Output

Paging the return of all rows in a table
This example uses the Chicago Ward Office table.  The code prints the contents for the Ward Number column for each of the rows.  Note that the first row is 0.

Code

from urllib2 import urlopen
from json import load

hostname = "data.cityofchicago.org"
viewID = "htai-wnw4"
startRow = 0
numRows = 20

while True:
    url = "http://%s/api/views/%s/rows.json?method=getRows&start=%s&length=%s" % (hostname, viewID, str(startRow), str(numRows))
    response = urlopen(url)
    rows = load(response)
    numRowsRcvd = len(rows)
    if numRowsRcvd == 0:
        break
    print '\n====== %s ROWS DOWNLOADED:  ======' % (numRowsRcvd)
    for row in rows:
        print row['2609304']      # rowID for ward number
    startRow += numRow

Output

Summary
Receiving your data in pages might make it easier for your app.  You may want to show, say, 20 results at a time.  Or, you may want to show the first 20 while you are getting the rest of the results.

For getting all the views in a site, you need to use paging if you there are more than 200.  The default is to receive 50 views.    You can use a parameter, XXX, to get 200 in one call.  Beyond that, you have to page to get all the views.