{"id":772,"date":"2020-04-30T22:03:02","date_gmt":"2020-05-01T02:03:02","guid":{"rendered":"http:\/\/www.aibistin.com\/?p=772"},"modified":"2023-03-12T16:04:22","modified_gmt":"2023-03-12T20:04:22","slug":"nyc-covid-infections-by-zip-code-with-python","status":"publish","type":"post","link":"https:\/\/www.aibistin.com\/?p=772","title":{"rendered":"NYC Covid Infections By Zip Code With Python"},"content":{"rendered":"\n

In my last post<\/a> I created a CLI tool to display NYC Covid-19 test results by Zip code using Perl, my favorite language for the moment. I would also like to do the same using Python. Purely as a an excuse to learn Python. This will download the same data, from the NYC health department’s GitHub page  <\/a>, and create a JSON file which I can use as a very basic database for later analysis. <\/p>\n

Here is an sample of the downloaded raw data.<\/p>\n\n\n\n

\"MODZCTA\",\"Positive\",\"Total\",\"zcta_cum.perc_pos\"\nNA,1558,1862,83.67\n\"10001\",309,861,35.89\n\"10002\",870,2033,42.79\n\"10003\",396,1228,32.25\n\"10004\",27,85,31.76\n\"10005\",54,206,26.21\n\"10006\",21,91,23.08\n\"10007\",49,204,24.02\n\"10009\",607,1745,34.79<\/code><\/pre>\n\n\n\n
This is the first iteration of my script.<\/p>\n\n\n
\nfrom __future__ import print_function\nimport datetime, json, requests, os, re, sys\n\n\nRAW_ZCTA_DATA_LINK = 'https:\/\/raw.githubusercontent.com\/nychealth\/coronavirus-data\/master\/tests-by-zcta.csv'\n\nALL_ZCTA_DATA_CSV = 'all_zcta_data.csv'\n\n# -------------------------------------------------------------------------------------------------\n#         Functions\n# -------------------------------------------------------------------------------------------------\ndef get_today_str():\n    today = datetime.date.today().strftime("%Y%m%d")\n    return today\n\ndef find_bin():\n    this_bin = os.path.abspath(os.path.dirname(__file__))\n    return this_bin\n\ndef create_dir_if_not_exists(base_dir, dir_name):\n    the_dir =  base_dir + '\/' + dir_name\n    if not os.path.isdir(the_dir):\n        os.mkdir(the_dir)\n    return the_dir\n\ndef create_db_dirs():\n    this_bin = find_bin()\n    db_dir = create_dir_if_not_exists(this_bin, 'db')\n    today_str = get_today_str()\n    year_month = today_str[0:4] + '_' + today_str[4:6];\n    year_month_dir = create_dir_if_not_exists(db_dir, year_month)\n    return year_month_dir\n\ndef get_covid_test_data_text():\n    r = requests.get(RAW_ZCTA_DATA_LINK)\n    print("Resp: " + str(r.status_code))\n    return r.text\n\ndef create_list_of_test_data():\n    test_vals = []\n    covid_text = get_covid_test_data_text()\n    for l in covid_text.splitlines():\n        lvals = re.split('\\s*,\\s*', l )\n        if lvals[0] == '"MODZCTA"':\n            continue\n        zip_dic = { 'zip' : lvals[0], 'positive': lvals[1],  'total_tested': lvals[2], 'cumulative_percent_of_those_tested': lvals[3]}\n        test_vals.append(zip_dic) \n    return test_vals\n\ndef write_todays_test_data_to_file():\n    year_month_dir = create_db_dirs()\n    test_data = create_list_of_test_data()\n    print(test_data[0])\n    today_str = get_today_str()\n    todays_file = year_month_dir + '\/' + today_str + '_tests_by_ztca.json'\n    out_file = open ( todays_file, 'w')\n    json.dump(test_data, out_file, indent=2)\n    print("Created todays ZTCA tests file,{todays_file}".format(**locals()))\n    out_file.close()\n\n\n\n# -------------------------------------------------------------------------------------------------\n\n\nwrite_todays_test_data_to_file()\n\n<\/pre><\/div>\n\n\n
Just a few snippets of interesting code here.<\/p>\n
To get todays date as a string in the format ‘yyyymmdd’, example, 20200401, I used the datetime module.<\/p>\n\n\n\n
today = datetime.date.today().strftime(\"%Y%m%d\")<\/code><\/pre>\n\n\n\n
Python has an interesting syntax for slicing strings or lists up into pieces. I used it here to create a directory name using the current year and month.<\/p>\n\n\n\n
year_month = today_str[0:4] + '_' + today_str[4:6]<\/code><\/pre>\n\n\n\n
The ‘[0:4]’ gets the first four characters of the string. The ‘[4:6]’ grabs the subsequent 2 characters of the string.  These are combined to create a sub-directory name like ‘2020_05’.<\/p>\n
To get the directory location of this script, kind-of similar to the Find::Bin in Perl, I used the path method of the os path library<\/a>. <\/p>\n\n\n\n
this_bin = os.path.abspath(os.path.dirname(__file__))<\/code><\/pre>\n\n\n\n
After downloading the raw test data for the current date from the NYC department of health GitHub page<\/a>, using the requests library<\/a>.<\/p>\n\n\n\n
r = requests.get(RAW_ZCTA_DATA_LINK)\n    print(\"Resp: \" + str(r.status_code))\n    return r.text<\/code><\/pre>\n\n\n\n
It is then split up using the ‘re’ module<\/a>, which seems to  be Pythons rather awkward way of doing regular expression matching.<\/p>\n\n\n\n
 lvals = re.split('\\s*,\\s*', l )<\/code><\/pre>\n\n\n\n
This will split each line of input data similar to this,<\/p>\n\n\n\n
\"10003\",396,1228,32.25<\/code><\/pre>\n\n\n\n
 Which can then be inserted to a python Dictionary structure like this,<\/p>\n\n\n
\n{\n  "zip": "10003",\n  "yyyymmdd": "20200503",\n  "positive": "396",\n  "total_tested": "1228",\n  "cumulative_percent_of_those_tested": "32.25"\n}\n<\/pre><\/div>\n\n\n
{\n  \"zip\": \"10003\",\n  \"yyyymmdd\": \"20200503\",\n  \"positive\": \"396\",\n  \"total_tested\": \"1228\",\n  \"cumulative_percent_of_those_tested\": \"32.25\"\n}<\/code><\/pre>\n\n\n\n
This is appended to the end of a list of similar Dictionaries.<\/p>\n\n\n\n
You may notice how I create the file path string is a little kludgy.<\/p>\n\n\n\n
 todays_file = year_month_dir + '\/' + today_str + '_tests_by_ztca.json'<\/code><\/pre>\n\n\n\n
I have since learned that there’s a better way to do this using the os path library<\/a>, which I’ll do the next time. <\/p>\n
To print the data in JSON format to a file, Python provides the aptly named ‘json’ library.  To dump the data to a file, simply,<\/p>\n\n\n\n
json.dump(test_data, out_file, indent=2)<\/code><\/pre>\n\n\n\n
The “indent=2”, isn’t necessary, but it makes the output more readable.<\/p>\n
To read JSON data from the file, <\/p>\n\n\n\n
test_data = json.load(in_file)<\/code><\/pre>\n\n\n\n
Read more about it here, Python JSON docs<\/a>.<\/p>\n
In the next post I will add more functionality to add more location details for each zip code where the tests were conducted, using a NYC Zip Code database<\/a> file.<\/p>\n\n\n\n\n\n\n\n\n\n
Adding some more functionality to merge the test data with the New York City Zip data to provide more details about each location where the tested people are from .<\/p>\n\n\n
\nimport datetime\nimport json\nimport requests\nimport os\nimport re\nimport sys\n\nRAW_ZCTA_DATA_LINK = 'https:\/\/raw.githubusercontent.com\/nychealth\/coronavirus-data\/master\/tests-by-zcta.csv'\nALL_ZCTA_DATA_CSV = 'all_zcta_data.csv'\nBIN_DIR = os.path.abspath(os.path.dirname(__file__))\nDB_DIR = os.path.join(BIN_DIR, '..', 'db')\nNA_ZIP = "88888"\nTHIS_SCRIPT = sys.argv[0]\nZIP_DB = os.path.join(DB_DIR, 'zip_db.json')\n...\n...\ndef get_zip_data():\n    z_db = open(ZIP_DB, 'r')\n    zip_data = json.load(z_db)\n    z_db.close()\n    return zip_data\n\n\ndef get_filler_location_rec(bad_zip=NA_ZIP):\n    label = 'Unknown-' + bad_zip\n    return {\n        'zip': bad_zip,\n        'borough': label,\n        'city': label,\n        'district': label,\n        'county': label\n    }\n\n\n# Zip: 11697\n# Data: {"borough": "Queens", "city": "Breezy Point", "county": "Queens", "district": "Rockaways"}\n# {\n#  "zip": "11697",\n#  "positive": "82",\n#  "total_tested": "193",\n#  "cumulative_percent_of_those_tested": "42.49"\n# }\n\ndef merge_zip_data():\n    all_zip_data = get_zip_data()\n    todays_test_data = get_todays_test_data()\n    merged_data = []\n\n    for td in todays_test_data:\n        if td['zip'] == 'MODZCTA':\n            continue\n\n        zip_data = all_zip_data.get(td['zip'])\n        if not zip_data:\n            print("NO Zip data for " + td['zip'])\n            zip_data = get_filler_location_rec(td['zip'])\n\n        zip_data.update(td.copy())\n        merged_data.append(zip_data)\n\n    return merged_data\n\n\ndef sort_test_data_func(zip_data):\n    return int(zip_data['positive'])\n\n\ndef write_todays_data_to_csv():\n    merged_test_data = merge_zip_data()\n    csv_file = get_todays_csv_file()\n    col_headers = [\n        'Zip',\n        'Date',\n        'City',\n        'District',\n        'Borough',\n        'Total Tested',\n        'Positive',\n        '% of Tested']\n    cols = [\n        'zip',\n        'yyyymmdd',\n        'city',\n        'district',\n        'borough',\n        'total_tested',\n        'positive',\n        'cumulative_percent_of_those_tested']\n\n    merged_test_data_sorted = sorted( merged_test_data, key=sort_test_data_func, reverse=True)\n   # merged_test_data_sorted = merged_test_data\n\n    csv_fh = open(csv_file, 'w')\n    csvwriter = csv.DictWriter(csv_fh, fieldnames=cols, restval='')\n    csvwriter.writeheader()\n    for zip_test_data in merged_test_data_sorted:\n        ztd = {x: zip_test_data[x] for x in cols}\n        csvwriter.writerow(ztd)\n\n    csv_fh.close()\n    print("Finished writing to the " + csv_file)\n\n\n<\/pre><\/div>\n\n
\nThe first subroutine, 'get_zip_data' just reads in the ZipCode data using Pythons 'json' library. The data comes in this format: \n<\/pre><\/div>\n\n\n
[\n  {\n    \"zip\": \"88888\",\n    \"yyyymmdd\": \"20200504\",\n    \"positive\": \"1607\",\n    \"total_tested\": \"1917\",\n    \"cumulative_percent_of_those_tested\": \"83.83\"\n  },\n  {\n    \"zip\": \"10001\",\n    \"yyyymmdd\": \"20200504\",\n    \"positive\": \"311\",\n    \"total_tested\": \"878\",\n    \"cumulative_percent_of_those_tested\": \"35.42\"\n  },\n...<\/code><\/pre>\n\n\n\n
The ‘get_filler_location_rec’ function adds default data in the case where sometimes the NYC department of health doesn’t provide the ZipCode for the test set. <\/p>\n
The test data and the ZipCode information is then merged to add more locations details to the test results.  It loops through the test data results, and for each record it does this,<\/p>\n\n\n\n
zip_data = all_zip_data.get(td['zip'])\n...\nzip_data.update(td.copy())<\/code><\/pre>\n\n\n\n
It gets a dictionary of the location information for the zip code, updates that with a copy of the test data for that location.  The combined Dictionary is then appended to the list of merged test data.<\/p>\n\n\n\n
merged_data.append(zip_data)<\/code><\/pre>\n\n\n\n
  Writing out the merged data to a CSV file can be done using the csv<\/a> library.  Id like to sort it in order of the number of positive cases descending,  using the ‘sorted’ function and applying my sort criteria with<\/p>\n\n\n\n
\ndef sort_test_data_func(zip_data):\n    return int(zip_data['positive'])\n...\n...\nmerged_test_data_sorted = sorted( merged_test_data, key=sort_test_data_func, reverse=True)<\/code><\/pre>\n\n\n\n
The resulting CSV file was double spaced.  So, after looking at the ‘csv’ library docs I changed the file open statement from:<\/p>\n\n\n\n
csv_fh = open(csv_file, 'w')<\/code><\/pre>\n\n\n\n
to<\/p>\n\n\n\n
csv_fh = open(csv_file, 'w', newline = '')<\/code><\/pre>\n\n\n\n
You’ll also notice, ( if you haven’t fallen asleep already) that I used the ‘csv.DictWriter’ (I’d love to know who comes up the naming in Python), as I am writing a list of Dictionaries to the CSV file. The DictWriter knows which dictionary fields to write to the CSV file using the “fieldnames=cols” attribute.\u00a0<\/p>\n
The CSV file looks something like this.<\/p>\n\n\n\n
\"\"
NYC Covid-19 testing data CSV<\/figcaption><\/figure>\n\n\n\n\n\n
20200509_tests_by_ztca<\/a>Download<\/a><\/div>\n\n\n\n
And that’s all I have to say about that.<\/p>\n","protected":false},"excerpt":{"rendered":"
In my last post I created a CLI tool to display NYC Covid-19 test results by Zip code using Perl, my favorite language for the moment. I would also like to do the same using Python. Purely as a an excuse to learn Python. This will download the same data, from the NYC health department’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[105,104,103,99,100],"tags":[59,71,34,48,95,58,73,68,69,72,70,74,50],"class_list":["post-772","post","type-post","status-publish","format-standard","hentry","category-csv","category-data","category-new-york-city","category-programming","category-python","tag-covid-19","tag-dictionary","tag-file","tag-json","tag-new-york-city","tag-nyc","tag-os-path","tag-python","tag-python3","tag-re","tag-requests","tag-tests","tag-zipcodes"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/posts\/772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=772"}],"version-history":[{"count":27,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/posts\/772\/revisions"}],"predecessor-version":[{"id":901,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=\/wp\/v2\/posts\/772\/revisions\/901"}],"wp:attachment":[{"href":"https:\/\/www.aibistin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aibistin.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}