The Idea
So I’ve had this idea for a mobile app kicking around for quite a while. It was born out of my time I used nerdery to buy a PlayStation 3 on the cheap project, where I leveraged some automation I created to search classified listings based on my criteria. If it found what I was interested in, it would send me email alerts.
The idea was to create a mobile app to do the same kind of thing – allow you to define some searches and pop a notification on your phone when it found something for you to look at so you could jump on the good deals before the next guy would steal them from you. I did a little research and, no surprise, that kind of app already existed for craigslist.
But I was living in Utah at the time, and while craigslist dominates the US classifieds market in almost every state, they don’t in Utah, where it’s all KSL’s classifieds. KSL-only would be a much smaller market, but I wasn’t really interested in the project for fame and fortune – it was an idea for something I would actually use, plus it seemed like a fun and useful idea to use as an excuse to get into mobile app development.
The Problem
Searching didn’t turn up anything existing along the lines of a KSL classifieds alerts app on any platform, so I set out to figure out how to make it happen. I checked to see if KSL had a public API, but that didn’t turn up anything, which wasn’t a big surprise. My PS3 project made full website page requests to ksl.com and knew how to parse out the data in which I was interested. Making repeated full page requests from a mobile app would be a bad idea – it would be much slower than an API request, not to mention a bandwidth hog, and nobody wants that – but it was looking like it might be my only option.
But then I had the thought – KSL’s official classified app, which didn’t really do anything beyond letting you browse the classifieds just like the website, had to be accessing the ad data via an API. If only I could monitor the requests that the app was making to see how it was getting its data…
After some Googling about how to monitor HTTP traffic for my phone, I came across the solution involving Fiddler, a free web debugging proxy. I’d used Fiddler previously to help debug web applications and create automation for website load testing, etc., so I was familiar with the tool. With it you can view all of the HTTP requests that your computer is making and inspect them – view the response, look at the headers, etc. At any rate, the solution was to set up Fiddler on my computer, then route my phone’s traffic through the Fiddler proxy server on my computer, which I did.
The Breakthrough
And what do you know – KSL’s app was making requests to:
- http://www.ksl.com/classifieds/api.php
An API! And not only that, it was completely unsecured – no HTTPS, no authentication of any kind required. I could take the requests I found in Fiddler made by the KSL app, run them in a browser, and get lovely JSON responses.
The next step was to figure out what API calls were available, so I started the process of going through each of the types of actions that the app contained and documenting it all (see full API documentation below).
How the Story Ends
After discovering the API, I was pretty excited about the project, even though it had a couple of limitations (no jobs or cars data, which are uniquely-handled sections of the site). I bought a few domains related to the idea, started working on the app design, started learning Android development, etc.
But then life happened – a move to a new state, a new job that required a lot of time, etc. and that was as far as the project ended up going.
I’ve had intentions to get back to it, but once I sat down and looked at it again last year, I found that KSL had moved up in the technology world – their classifieds app was now using a version 2 of their API that used HTTPS and authentication at api2.ksl.com (and now they’ve upgraded to version api3.ksl.com).
A new API request, like this one recorded in Fiddler looks like it includes authentication tokens and gives you this response if you try and request it directly:
- Invalid request: authentication failed
I figured that the old, original, non-secure API that I’d found would be shut down at that point, which would ruin my project, but it wasn’t – it was still there and working. It’s been about another year and it’s still up. But, even though the old API is still up, I’ve basically abandoned hope that I’ll see the project through, so I figured I should at least document what I found so that if a better man (or woman) comes along, they can benefit from my findings. At least the initial discovery phase was pretty fun and educational.
KSL Classifieds API Documentation
Base API URL:
- http://www.ksl.com/classifieds/api.php
General Parameters
- cmd
- description:
- command
- values:
- categories
- list all subcategories (includes parent category w/o parent ID)
- list
- list category or search items
- ad
- show individual classified ad
- categories
- description:
- c
- description:
- number of results
- values:
- integer
- description:
- o
- description:
- starting point for results
- values:
- integer
- description:
- nid
- description:
- category id
- values:
- integer
- description:
- id
- description:
- ad id
- values:
- integer
- description:
Search Parameters
- s
- description:
- search string
- values:
- string
- description:
- min
- description:
- minimum amount
- values:
- int
- description:
- max
- description:
- maximum amount
- values:
- int
- description:
- d
- description:
- distance (in miles)
- values:
- int
- description:
- z
- description:
- zip code
- values:
- string
- description:
- srt
- description:
- sort order
- values:
- Most+recent
- Least+recent
- Lowest+price
- Highest+price
- description:
- slr
- description:
- seller type
- values:
- All+sellers
- Private+sellers
- Businesses
- description:
Example Requests and Responses
* Note that the data and in these examples is old (from around March 2014).
- List all categories (and subs)
- URL
- http://www.ksl.com/classifieds/api.php?cmd=categories
- return (multiple):
- “id”:”2″,
- “category”:”Announcements”,
- “subcategory”:”Charity”,
- “count”:”201″
- URL
- List category or subcategory items
- URL
- http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&nid=7
- returns (multiple):
- “sid”:”23238199″,
- “nid”:”2″,
- “title”:”A simple way to help families hoping to adopt”,
- “price”:”0.00″,
- “image”:”http:\/\/img.ksl.com\/c\/2567\/256784\/25678426.png”,
- “display_time”:1354598428,
- “city”:”Farmington”,
- “state”:”UT”
- URL
- Search
- URL
- http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&s=searchterm&min=1&max=9&d=50&z=84653&srt=Most+recent&slr=All+sellers
- returns (multiple):
- “sid”:”23147173″,
- “nid”:”91″,
- “title”:”Full size, 30 inch, Wheeled Duffel Bag- brand NEW, was a Black Friday special, changed my mind! Retail: $99″,
- “price”:”45.00″,
- “image”:”http:\/\/img.ksl.com\/c\/2546\/254680\/25468076.jpg”,
- “display_time”:1354060300,
- “city”:”Provo”,
- “state”:”UT”
- URL
- Search in specific category (just add nid param)
- URL
- http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&s=house&min=1&max=9&d=50&z=84653&srt=Most+recent&slr=All+sellers&nid=279
- returns (multiple):
- “sid”:”22825446″,
- “nid”:”279″,
- “title”:”Beautiful House for Sale in Idaho Falls, ID”,
- “price”:”169000.00″,
- “image”:”http:\/\/img.ksl.com\/c\/2472\/247266\/24726676.jpg”,
- “display_time”:1354597922,
- “city”:”Idaho Falls”,
- “state”:”ID”
- URL
- Show individual ad
- URL
- http://www.ksl.com/classifieds/api.php?cmd=ad&id=23027643
- return:
- “sid”:”23238150″,
- “category”:”94″,
- “subcategory”:”640″,
- “title”:”32 ft ext ladder”,
- “price”:”$175.00″,
- “displaytime”:”1354598118″,
- “sold”:””,
- “body”:”32 ft. extension ladder $175.00 firm\nKeller type 1 Industrial Aluminum ladder”,
- “contactname”:”Dorothy”,
- “contacthomephone”:”801-201-5615″,
- “contactworkphone”:”801-201-5615″,
- “contactcellphone”:””,
- “city”:”West jordan”,
- “state”:”UT”,
- “images”:[
- “http:\/\/img.ksl.com\/c\/2567\/256787\/25678721.jpg”,
- “http:\/\/img.ksl.com\/c\/2567\/256787\/25678779.jpg”,
- “http:\/\/img.ksl.com\/c\/2567\/256789\/25678901.jpg”
- ],
- “pageviews”:”5″
- URL
- Large image
- http://img.ksl.com/c/2566/256635/25663529.jpg?filter=classified/mobile_adPic1
- Small image:
- http://img.ksl.com/c/2566/256635/25663529.jpg?filter=ksl/newhl
This is awesome. I’ve been thinking about a project this for a while now. Thank you, thank you, thank you for this information. I’m sad that I didn’t realize there must be an API for their app and reverse engineer it, but this makes the whole thing feasible.
@Brandon – I hope it’s useful for you, but no guarantees. 🙂 It’s been quite a while since I created the original documentation – probably at least 3 years – so between that fact and the fact that it’s v1 of their API that they’re on at least v3 of, results might not be great. One person that contacted me recently about it said they were having problems getting filters, etc. to work.
Preliminary testing looks good, but I haven’t gone in depth yet. This is kind of like your project to me, just fun and a cool learning experience.
Awesome. Let me know how it turns out.
Pingback: » Use Bash to Automate Checking for that Dream Job
God bless you
It appears that the API is now offline. Attempts to query it result in “Page not found” error responses. 🙁 Honestly I think KSL should simply build in a saved search notification system. It wouldn’t be hard for them to do, and it would be event driven, as any in coming ad or update would get ran against the saved queries. Verses using a polling method as discussed here. Imagine how that’s going to scale as you have thousands of users, with potentially hundreds of searches, being ran every X number of minutes… Really not the way to do this. I think if someone were to publish an app/site KSL would probably reactive negatively to it due to the load it would place on their system.
I too have been thinking about doing this for some time! Thank you so much for posting this, you are famous in my book!