discovering and documenting the unpublished ksl classifieds api

The Idea

So I’ve had this idea for a mobile app kicking around for quite a while. It was born out of my time I used nerdery to buy a PlayStation 3 on the cheap project, where I leveraged some automation I created to search classified listings based on my criteria. If it found what I was interested in, it would send me email alerts.

The idea was to create a mobile app to do the same kind of thing – allow you to define some searches and pop a notification on your phone when it found something for you to look at so you could jump on the good deals before the next guy would steal them from you. I did a little research and, no surprise, that kind of app already existed for craigslist.KSL Classifieds

But I was living in Utah at the time, and while craigslist dominates the US classifieds market in almost every state, they don’t in Utah, where it’s all KSL’s classifieds. KSL-only would be a much smaller market, but I wasn’t really interested in the project for fame and fortune – it was an idea for something I would actually use, plus it seemed like a fun and useful idea to use as an excuse to get into mobile app development.

The Problem

Searching didn’t turn up anything existing along the lines of a KSL classifieds alerts app on any platform, so I set out to figure out how to make it happen. I checked to see if KSL had a public API, but that didn’t turn up anything, which wasn’t a big surprise. My PS3 project made full website page requests to ksl.com and knew how to parse out the data in which I was interested. Making repeated full page requests from a mobile app would be a bad idea – it would be much slower than an API request, not to mention a bandwidth hog, and nobody wants that – but it was looking like it might be my only option.

But then I had the thought – KSL’s official classified app, which didn’t really do anything beyond letting you browse the classifieds just like the website, had to be accessing the ad data via an API. If only I could monitor the requests that the app was making to see how it was getting its data…

Fiddler LogoAfter some Googling about how to monitor HTTP traffic for my phone, I came across the solution involving Fiddler, a free web debugging proxy.  I’d used Fiddler previously to help debug web applications and create automation for website load testing, etc., so I was familiar with the tool. With it you can view all of the HTTP requests that your computer is making and inspect them – view the response, look at the headers, etc. At any rate, the solution was to set up Fiddler on my computer, then route my phone’s traffic through the Fiddler proxy server on my computer, which I did.

The Breakthrough

And what do you know – KSL’s app was making requests to:

  • http://www.ksl.com/classifieds/api.php

An API! And not only that, it was completely unsecured – no HTTPS, no authentication of any kind required. I could take the requests I found in Fiddler made by the KSL app, run them in a browser, and get lovely JSON responses.

Fiddler Output

(these are actually requests to the new, secured API – I neglected to screenshot requests to the old API)

The next step was to figure out what API calls were available, so I started the process of going through each of the types of actions that the app contained and documenting it all (see full API documentation below).

How the Story Ends

After discovering the API, I was pretty excited about the project, even though it had a couple of limitations (no jobs or cars data, which are uniquely-handled sections of the site). I bought a few domains related to the idea, started working on the app design, started learning Android development, etc.

But then life happened – a move to a new state, a new job that required a lot of time, etc. and that was as far as the project ended up going.

I’ve had intentions to get back to it, but once I sat down and looked at it again last year, I found that KSL had moved up in the technology world – their classifieds app was now using a version 2 of their API that used HTTPS and authentication at api2.ksl.com (and now they’ve upgraded to version api3.ksl.com).

A new API request, like this one recorded in Fiddler looks like it includes authentication tokens and gives you this response if you try and request it directly:

  • Invalid request: authentication failed

I figured that the old, original, non-secure API that I’d found would be shut down at that point, which would ruin my project, but it wasn’t – it was still there and working. It’s been about another year and it’s still up. But, even though the old API is still up, I’ve basically abandoned hope that I’ll see the project through, so I figured I should at least document what I found so that if a better man (or woman) comes along, they can benefit from my findings. At least the initial discovery phase was pretty fun and educational.

KSL Classifieds API Documentation

Base API URL:

  • http://www.ksl.com/classifieds/api.php

General Parameters

  • cmd
    • description:
      • command
    • values:
      • categories
        • list all subcategories (includes parent category w/o parent ID)
      • list
        • list category or search items
      • ad
        • show individual classified ad
  • c
    • description:
      • number of results
    • values:
      • integer
  • o
    • description:
      • starting point for results
    • values:
      • integer
  • nid
    • description:
      • category id
    • values:
      • integer
  • id
    • description:
      • ad id
    • values:
      • integer

Search Parameters

  •  s
    • description:
      • search string
    • values:
      • string
  • min
    • description:
      • minimum amount
    • values:
      • int
  • max
    • description:
      • maximum amount
    • values:
      • int
  • d
    • description:
      • distance (in miles)
    • values:
      • int
  • z
    • description:
      • zip code
    • values:
      • string
  • srt
    • description:
      • sort order
    • values:
      • Most+recent
      • Least+recent
      • Lowest+price
      • Highest+price
  • slr
    • description:
      • seller type
    • values:
      • All+sellers
      • Private+sellers
      • Businesses

Example Requests and Responses

* Note that the data and in these examples is old (from around March 2014).

  • List all categories (and subs)
    • URL
      • http://www.ksl.com/classifieds/api.php?cmd=categories
    • return (multiple):
      • “id”:”2″,
      • “category”:”Announcements”,
      • “subcategory”:”Charity”,
      • “count”:”201″
  • List category or subcategory items
    • URL
      • http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&nid=7
    • returns (multiple):
      • “sid”:”23238199″,
      • “nid”:”2″,
      • “title”:”A simple way to help families hoping to adopt”,
      • “price”:”0.00″,
      • “image”:”http:\/\/img.ksl.com\/c\/2567\/256784\/25678426.png”,
      • “display_time”:1354598428,
      • “city”:”Farmington”,
      • “state”:”UT”
  • Search
    • URL
      • http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&s=searchterm&min=1&max=9&d=50&z=84653&srt=Most+recent&slr=All+sellers
    • returns (multiple):
      • “sid”:”23147173″,
      • “nid”:”91″,
      • “title”:”Full size, 30 inch, Wheeled Duffel Bag- brand NEW, was a Black Friday special, changed my mind! Retail: $99″,
      • “price”:”45.00″,
      • “image”:”http:\/\/img.ksl.com\/c\/2546\/254680\/25468076.jpg”,
      • “display_time”:1354060300,
      • “city”:”Provo”,
      • “state”:”UT”
  • Search in specific category (just add nid param)
    • URL
      • http://www.ksl.com/classifieds/api.php?cmd=list&c=50&o=0&s=house&min=1&max=9&d=50&z=84653&srt=Most+recent&slr=All+sellers&nid=279
    • returns (multiple):
      • “sid”:”22825446″,
      • “nid”:”279″,
      • “title”:”Beautiful House for Sale in Idaho Falls, ID”,
      • “price”:”169000.00″,
      • “image”:”http:\/\/img.ksl.com\/c\/2472\/247266\/24726676.jpg”,
      • “display_time”:1354597922,
      • “city”:”Idaho Falls”,
      • “state”:”ID”
  • Show individual ad
    • URL
      • http://www.ksl.com/classifieds/api.php?cmd=ad&id=23027643
    • return:
      • “sid”:”23238150″,
      • “category”:”94″,
      • “subcategory”:”640″,
      • “title”:”32 ft ext ladder”,
      • “price”:”$175.00″,
      • “displaytime”:”1354598118″,
      • “sold”:””,
      • “body”:”32 ft. extension ladder $175.00 firm\nKeller type 1 Industrial Aluminum ladder”,
      • “contactname”:”Dorothy”,
      • “contacthomephone”:”801-201-5615″,
      • “contactworkphone”:”801-201-5615″,
      • “contactcellphone”:””,
      • “city”:”West jordan”,
      • “state”:”UT”,
      • “images”:[
        • “http:\/\/img.ksl.com\/c\/2567\/256787\/25678721.jpg”,
        • “http:\/\/img.ksl.com\/c\/2567\/256787\/25678779.jpg”,
        • “http:\/\/img.ksl.com\/c\/2567\/256789\/25678901.jpg”
      • ],
      • “pageviews”:”5″
  • Large image
    • http://img.ksl.com/c/2566/256635/25663529.jpg?filter=classified/mobile_adPic1
  • Small image:
    • http://img.ksl.com/c/2566/256635/25663529.jpg?filter=ksl/newhl
Posted in technical

hiring using careers 2.0 by stackoverflow

Careers 2.0 by StackOverflow

I’ve got invitations to create a job seeker profile on Careers 2.0 available for anyone interested – just let me know.

Things at the company are going well. You’ve got too much work to do and not enough people. You decide you need to hire another developer or two so you can maintain what you’ve already got going on plus go after those exciting new ideas.

Options

So where do you start?

Networking

Ideally you rely on some networking. Talk to people you (and your co-workers) have worked with in the past that you’d love to work with again. Get referrals from them if they’re not interested themselves. Talk to friends and family. Networking got us one great developer that I’d worked with for years at a previous company (technically not a finalized deal at time of writing, but dangerously close). Familiarity with his background – knowledge, work ethic, etc. – plus a solid track record of having worked together successfully in the past is absolutely huge as you’re dealing with a known entity (a strong reference from someone you know and respect is a strong second).

Technical Recruiters

No thanks, if I can help it.

  • Ridiculous finders fees
    • 25-35% of the candidate’s first year salary? Really? Does that seem proportional to anyone? Is it just me? Yeah we can afford to pay it but…geez.
  • Inability (or lack of desire) to filter candidates
    • Throw enough crap against the wall and something is bound to stick, right? Too many recruiters are either too ignorant of the technology sector they work in to be able to discern qualifications, or they just don’t care. Either way it results in wasted time for the side doing the hiring (not to mention the candidates).
  • Spam tactics
    • I get so tired of job spam emails from tech recruiters. I haven’t updated any online job search profile in forever and they still spam me, sometimes from information in their “database” that is 5+ years old. They might recognize a buzzword on your resume and suddenly they think you must be a great fit for this job they’re looking to fill. They do absolutely no research on your background, current situation, etc. They know nothing about you but if they spam enough, someone might actually respond. Who cares if your response rate is 1 out of 100 if you’ve got the ability to send near-unlimited amounts of spam? I don’t want to specifically call anyone out here but… **cough**CyberCoders is the worst**cough**

Online Job Boards

You’ve got the older school sites – Monster, CareerBuilder, Craigslist, etc. I haven’t found them to be worth much. Dice has been around a while and is specifically for tech jobs, but most of the recruiter spam I get comes from recruiters who found an old profile on Dice, which is a pretty big turnoff. There’s Indeed. LinkedIn isn’t a bad option. And then there’s a relatively new contender – Careers 2.0 by StackOverflow.

Careers 2.0 by StackOverflow

If you’re a developer, you know StackOverflow. You live by StackOverflow. Community-created and maintained Q&A format programming content. A gamification reputation system that encourages involvement and high-quality content (here’s my profile if you’re interested). It may be the single-greatest online programming resource known to man. When I’m Googling for information or how to solve a problem, I automatically skip down to the first StackOverflow result as my first option. It’s often the only option I need. A little over 3 years ago, StackOverflow released a job search product called Careers 2.0. Careers 2.0 by StackOverflow

The Good

The main bullet points from their launch blog post:

  1. Free to job seekers, but invite-only
  2. Better profiles – the intent to be the ultimate programmer’s portfolio, complete with integration to your StackOverflow profile.
  3. Support for passive candidates
  4. Better search – statistics and nice drill-down maps, etc.

In theory, Careers 2.0 is great. It combines a developer job search with the best online developer reference we have. Some other positives about their product from the employer side:

  • Relocation search – candidates can specify areas which they are open to relocate to as part of their profile, so you’re not limited to just the candidates in your area – you can potentially pull from other areas and actually know that someone is willing to relocate.
  • Decent tools for keeping track of your search – enter notes about candidates, move them along in a kanban-style board from one stage of the process to the next
  • Saved searches – create searches with different parameters and save them for re-use, with candidates found in those searches linked back to them for reference

The cost for access to their candidate search for employers is $1000 per month. It’s not chump change, but it’s not unreasonable, especially if their candidate content is very high quality.

The Bad

  • Small Las Vegas candidate pool
    • This one isn’t entirely Careers 2.0’s fault. Las Vegas is a decent-sized metro area, but is not a tech company mecca. As my former boss, who grew up in Vegas, put it – if you want to work in tech in Vegas you can either work for the government, the gambling industry, or Zappos. Incidentally, Tony Hsieh, the CEO of Zappo’s, is spearheading the Downtown Project, a revitalization effort for downtown Las Vegas with an entrepreneurial and technology emphasis. The number of dev candidates in the area pales in comparison with the area I moved away from a year ago – Salt Lake City/Provo/Orem – which is rapidly turning into a mini Silicon Valley.
  • Outdated profiles
    • Easily the single biggest problem with the service. An incredibly high percentage of the profiles matching my searches were outdated. And we’re not just talking a few months or a year – years with an ‘s’. For something that’s only been around for 3 years, to see so many abandoned profiles is kind of a big surprise. They have got to do something to deal with the problem – whether that’s letting you filter/sort by when a profile was updated, or a proactive effort to re-engage users whose profiles are way out of date…something.
    • LinkedIn became my best friend when using Careers 2.0, which is obviously pretty odd. I would find an interesting candidate, his/her profile would either be incomplete or very outdated, so I’d turn to Google and LinkedIn to find more info. Simply searching for “linkedin [candidate name]” usually provided me with the candidate’s LinkedIn profile, which 9 times out of 10 was way better than what I found on Careers 2.0.
      • Side note about LinkedIn and Google searches.
        • LinkedIn has premium account offerings that let you see more details about people not in your LinkedIn network. The better subscription you have, the more and farther out of your network you can see. I had a basic premium account that gave me some information but not all.
        • If you Google for someone and find a result for their LinkedIn profile and you’re logged into LinkedIn, the information you can see is often limited if they’re not in your network. But if you log out of LinkedIn, or simply run the search in another browser in which you’re not logged in, you’ll be able to see a much more complete profile than you could when you were logged in. I’ve been using that trick for quite a while and also came across an article where someone else mentioned the same tactic a couple months ago.
  • Poor response rates
    • This is likely largely a function of the outdated profile problem detailed above, but response rates, even to just get a “not interested” reply, were very poor. You have a limited number of inquiries you can have pending at any given time (it’s a pretty healthy number though), so some of your subscription time is wasted in just waiting for responses that may never come (and eventually time out).
  • Customer service
    • This one may have just been bad luck or an anomaly, but one day my Careers 2.0 rep sent me an email to check in and see how the job search was going. He asked for feedback. I told him about the shortcomings that I listed above. I never got a response. He did send me offers for free job postings a few times in the months following though.

Disclaimer

Despite its significant shortcomings, I have to give credit where credit is due. My current employer found me via Careers 2.0. I’m very grateful for that. And we eventually did find a great developer that we hired – and found him via Careers 2.0. I feel like it was more a stroke of good luck in that case rather than the result of an awesome product. The candidate we hired turned out to be the only one that even made it more than a couple steps into our hiring process and we lucked out that he was a great fit and was interested in joining us – but I’ll take it either way.

Posted in technical

the time I used nerdery to buy a ps3 on the cheap

MLB 11 The Show for PS3

MLB 11 The Show for PS3Three years ago, I got the itch to play some video game baseball. It had been probably been 4-5 years since I’d owned any baseball games and I determined that the time had come. After a little online research, I found that the consensus best baseball game was MLB 11 The Show (Joe Mauer was on the cover that year) and it was only available on the PlayStation 3. Problem was, I only had an Xbox 360. The only baseball game for Xbox 360 was MLB 2K11, which apparently was crap.

Well although I really wanted to play some baseball, there was no way I was going to buy a $300+ gaming console for just one game – I’m not a hardcore gamer and there just wasn’t any justifying that. Yeah, I technically had a second justification in that I still didn’t have a Blu-ray player and the PS3 could function in that role for me, but that still wasn’t sufficient – I’m way too cheap for that to be enough.

How About a Used PS3?

Since a brand new PS3 was out of the question due to price, I started looking at used ones in places like eBay, Craigslist, and KSL Classifieds (I was living in Utah at the time and KSL’s classifieds are much bigger than Craigslist there). I preferred a newer model as I intended to keep it a while for the Blu-ray aspect, and also in case any other PS3 games came along that I was interested in.

Every so often I would check one of the sites mentioned above to see if any good PS3 deals had popped up. And pop up they did – but I was always too late for the REALLY good deals – by the time I called on them they were long gone, even if they’d only been posted maybe 30 mins. before. After seeing some of the great deals come and go, usually on Craigslist and KSL, I was pretty set on waiting for a good deal for a used machine, but was feeling some frustration due to always being too slow. Basically, to even have a chance of catching a great deal, I had to constantly be checking Craigslist and KSL, and who has the time for that? If only there were a way to be alerted immediately as soon as something I was interested in was listed so I could beat out those other deal-seeking fools…

Nerdery to the Rescue

And I thought to myself, “Wait a minute, I’m a computer nerd – there’s got to be a computer nerd way to approach this.” A year or two before this, I had built a simple website monitor  to monitor the uptime of some sites I run – it requests a page, makes sure it loads with an HTTP 200 status code, and looks for a string of text on the page, sending an email alert if it fails. I figured  the same basic concept could easily work for doing my PS3 classified searches.

So I built a little classified watcher app for Windows. It allowed me to configure multiple searches against KSL and Craigslist:

<watcher 
    name="KslPs3" 
    url="http://www.ksl.com/index.php?nid=231&nocache=1&search=playstation&zip=&distance=&min_price=&max_price=&type=&x=29&y=11" 
    type="ksl" 
/>
<watcher 
    name="CraigsListProvoPs3"
    url="http://provo.craigslist.org/search/vga?query=ps3&minAsk=150&maxAsk=200"
    type="craigslist"
/>

The searches were configured by manipulating the URLs of the requests – min/max price, distance from me, etc. I made separate handlers for KSL and Craiglist due to the obvious differences in their sites. Every time the watcher would run, it would check the ads it found against the list of ads that I’d already seen in previous requests, and then send me email alerts with any new ads matching my criteria (if an ad was updated then it would get re-sent). I set it up to run every 10 mins. or so using Windows Task Scheduler.

My new classified watcher app ran for a few days without finding me anything awesome, but then I received this email one morning:

PS3 for Cheap Ad Email

Nice – a 120 GB PS3 with 16 games, 2 controllers, and the DJ Hero turntable. Even if the games were complete crap, which it turns out they weren’t (at least not all of them), that was a screaming deal. I called the guy, told him I’d give him his asking price, that I’d pick it up that morning, and he said it was mine. As it was a weekday, I still had to get ready for work, then the guy’s house was an extra 20 min. farther drive than work, so it was a bit before I got there. While I was on the way, I got this update from my classified watcher that helped reassure me that the deal was mine:

PS3 for Cheap Ad Email Sold

Makin’ Money

Amazon Game Trade-In Receipt

Amazon Game Trade-In Receipt

So the package came with 16 games, but I didn’t really want any of them, so I figured I’d just sell them off. I pulled up Amazon’s video game trade-in store to see what they were worth. Amazon’s trade-in program is pretty nice – they tell you up front how much they’ll give you for the games, then they’ll give you a UPS shipping label and pay for the games to be shipped to them. They also have trade-in stores for electronics, textbooks, DVDs, and music. Some of the games were worth just a few bucks, and I ended up keeping a couple of the decent ones due to that fact, but some were actually worth something.

Trade-in total: $157.75

Wrap-Up

Did my nerdery work and get me a PS3 for cheap? Yes it did. Was it worth it? Probably not, given the time and effort I put into finding one on the cheap, including the few hours it took to put together my classified watcher app, etc. But I’m fine eating that time cost as it was a fun little project that yielded the exact result that I wanted. When it was all said and done I think I ended up with:

  • 120 GB PS3
  • 2 controllers
  • Fallout 3
  • Battlefield: Bad Company
  • DJ Hero and turntable (tried to sell this for as little as $20 but no takers – I think we eventually threw it away after trying it like twice)

Grand Total: $42.25 (after game trade ins)

Posted in technical

moneyball book review

Moneyball

Confessions

MoneyballIn some ways, I was pretty late to the whole Moneyball thing.  I didn’t see the critically-acclaimed movie, released in late 2011, until well after it had been released on DVD. And the book that I recently finished was published in 2004, so I was about 8 years behind there.

But I’ve been familiar with some of the concepts central to the theme of Moneyball for quite some time. I was a pretty dedicated fantasy baseball nerd for years (I retired a few years ago – it was consuming too much of my time). In the pursuit of fantasy baseball championships, I turned to advanced baseball analysis and statistics, like those available on fangraphs.com.

Although the statistics normally used in fantasy baseball are based on traditional baseball statistics, I found that using sabermetric analysis could still help me tremendously in predicting future performance. It allowed me to look at players differently than most of the other people in my league, identifying players whose surface stats were either inflated or depressed compared to their actual skills and superior traditional statistics (often in very small sample sizes). With that valuation analysis I was able to consistently buy low/sell high, deriving substantial benefits (i.e., I almost always won).

Moneyball Is Not a Book About Baseball

Ok, maybe it is about baseball, technically. Billy Beane is a former baseball player turned general manager of the Oakland A’s, a small market team trying to compete on the field while being restricted by a payroll literally several times less than the richest teams. On base percentage (OBP), old school scouting vs. new age statistics, big market teams vs. small market teams, etc.

So it is about baseball on the surface. But it is replete with lessons that are applicable to business, and life in general. Here are a few of the themes that I identified while reading.

That’s Just the Way It’s Done

  • Question conventional wisdom
  • Don’t be too scared to fail

Applicable quote from the book:

“Managers tend to pick the strategy that is the least likely to fail, rather than to pick a strategy that is most efficient. The pain of looking bad is worse than the gain of making the best move.”

Will it get you results continuing to do things the way they’ve always been done? Maybe, but it’s not going to get you very far ahead. In order to do that, you’ve either got to perfect the conventional processes through increased efficiency, etc., or you’ve got to take a different approach. Think about things differently. Which leads to…

Ask the Right Questions

Baseball is simple. Score more runs than the other team and you win. Run production and run prevention. So:

  • wins = runs

But, traditionally, the line of thinking has been:

  • wins = players

Quote (from the movie):

Peter Brand: There is an epidemic failure within the game to understand what is really happening. And this leads people who run Major League Baseball teams to misjudge their players and mismanage their teams. I apologize.

Billy Beane: Go on.

Peter Brand: Okay. People who run ball clubs, they think in terms of buying players. Your goal shouldn’t be to buy players, your goal should be to buy wins. And in order to buy wins, you need to buy runs. You’re trying to replace Johnny Damon. The Boston Red Sox see Johnny Damon and they see a star who’s worth seven and half million dollars a year. When I see Johnny Damon, what I see is… is… an imperfect understanding of where runs come from. The guy’s got a great glove. He’s a decent leadoff hitter. He can steal bases. But is he worth the seven and half million dollars a year that the Boston Red Sox are paying him? No. No. Baseball thinking is medieval. They are asking all the wrong questions. And if I say it to anybody, I’m-I’m ostracized. I’m-I’m-I’m a leper. So that’s why I’m-I’m cagey about this with you. That’s why I… I respect you, Mr. Beane, and if you want full disclosure, I think it’s a good thing that you got Damon off your payroll. I think it opens up all kinds of interesting possibilities.

Not Everything (or Everyone) of Value Will Fit Your Expected Mold

Don’t judge a book by its cover.

“We’re not selling jeans here” is one of my favorite Beane quotes. Beane’s old school scouts love to fixate on the physical attributes of the players, using lines such as “The guy has a great body,” to which Beane responds “We’re not selling jeans here.”

Here’s one of the best exchanges, regarding a player named Jeremy Brown. Brown’s actual on-field collegiate production was outstanding but due to the fact that he, to put it politically incorrectly, was fat, he was completely ignored by all scouts:

“It’s soft body,” says the most vocal old scout. “A fleshy kind of body.”

“Oh, you mean like Babe Ruth?” says Billy. Everyone laughs, the guys on Billy’s side of the room more happily than the older scouts across from him.

“I don’t know,” says the scout. “A body like that can be low energy.”

“Sometimes low energy is just being cool,” says Billy.

“Yeah,” says the scout. “Well, in this case low energy is because when he walks, his thighs stick together.”

“I repeat: we’re not selling jeans here,” says Billy.

“That’s good,” says the scout. “Because if you put him in corduroys, he’d start a fire.”

  1. Do you get caught up in the fact that you don’t fit the mold, resulting in lack of effort and not rising to your potential?
  2. Do you limit others’ actual production or potentials with the same kind of thinking (and resulting actions) regarding them?

Identify Market Inefficiencies

Beane’s team was at a significant disadvantage due to being a small market team with low revenues, resulting in not having an abundance of money to spend on building the team. While the Yankees were spending $125+ million in yearly payroll, Oakland was spending  maybe $40 million. With the inability to throw large wads of cash at big name free agents, the team had to identify areas of the baseball market that were undervalued. This led to using sabermetrics to find:

  1. Which skills actually led to baseball production, i.e., winning games
  2. Which players possessed those skills but were ignored by other teams/scouts

“The inability to envision a certain kind of person doing a certain kind of thing because you’ve never seen someone who looks like him do it before is not just a vice. It’s a luxury. What begins as a failure of the imagination ends as a market inefficiency: when you rule out an entire class of people from doing a job simply by their appearance, you are less likely to find the best person for the job.”

Conclusion

bool youLikeBaseball = false;

Console.WriteLine("Enter 'y' if you like baseball. Enter 'n' if you do not");

if (Console.ReadLine() == "y")
{
    youLikeBaseball = true;
}

if (youLikeBaseball)
{
    Console.WriteLine("Read Moneyball. You'll gain a new and valuable perspective about which statistics in the sport are truly meaningful and how value in it should really be measured.");
}
else
{
    Console.WriteLine("Read Moneyball. It's an enjoyable read with a great context to help foster some serious thought about how you approach business and life.");
}
Posted in books

mac vs. pc

Disclaimer 1: This isn’t intended to be an in-depth, all-inclusive review/comparison of the two platforms – just some observations based on my experience, both software and hardware. I don’t claim to know everything about Macs or Mac OS at this point either – I’m sure I’m still ignorant of many of its useful and interesting features. 

Disclaimer 2: I’m not a big Apple guy. I generally dislike their walled garden approach to computing, although I acknowledge that they produce quality products admired by many. I’m convinced that they’re every bit as evil as Microsoft was in their prime though. I prefer Android over iOS (I like to think that I’m not an Android Nazi though – some of my kids have iPhones). That bias may be evident in what follows.

Intro

I’ve been a Windows user for a long time. That’s what I’ve been comfortable with and I’d never felt that I had much of a need to try out a Mac. The handful of times I’d been forced to use a Mac computer for short periods had been pretty frustrating – it was all different and weird. Not to mention the fact that their computers seemed pretty overpriced. Then, a little over a year ago, I had the option to start using a MacBook Pro laptop at work. I’d heard good things about the hardware, etc., my boss at the time, whom I respect greatly, is a hardcore Apple guy, and I decided that I could branch out a little – learn something new. So I gave it a shot because hey, I didn’t have to pay for it. 😉

At any rate, here are some thoughts on some different areas based on notes I’ve jotted down over the past year+. The good, the bad, the annoying.

Operating Systems in General

Both Windows and Mac OS are quality operating systems. I think Windows 7 is great (I have not tried Windows 8 for more than a few minutes at a time, so no opinion there). Mac OS isn’t bad at all, outside of poor window management (mentioned below) and Finder, which I think is pretty much awful (perhaps more on that in Part 2). You can get things done very well with either OS. Each has some pros and cons, but I don’t really think that one stands out significantly over the other. At this point, if I have to choose I’m still going with Windows though.

Keyboard Differences

Shortcut Hell

This was probably the most difficult part of the transition, and still sometimes gives me trouble when switching back and forth between Mac OS and Windows, although I’m a lot better at the back and forth now (I run a Windows VM inside of Mac OS and use Windows on my desktop at home). I’m a fairly heavy keyboard user – window switching, document navigation, searching within a document, copy/paste, opening/closing tabs, opening applications, etc. You can perform all of the same actions, but the shortcut keys/combinations are all different. Here are a few examples.

  • Switching windows
    • Windows = alt + tab
    • Mac = Command + tab
  • Find
    • Windows = control + F
    • Mac = Command + F
  • Copy/Paste
    • Windows = control + C, control + V
    • Mac = Command + C, Command + V
  • Close Application
    • Windows = alt + F4
    • Mac = Command + Q (+1 for Mac on this one – much nicer)
  • Close Tab
    • Windows = control + W
    • Mac = Command + W
  • Refresh Browser
    • Windows = F5 (and control + F5 to force refresh)
    • Mac = Command + R (no command to force, which sucks)
  • Open Application
    • Windows = Windows key + start typing name then hit Enter
    • Mac = Command + spacebar + start typing name then hit Enter (called Spotlight)
      • I use both of these methods almost exclusively to open apps. Spotlight generally feels like it filters more quickly.

I feel like Mac OS is generally more consistent, using Command + key pretty consistently whereas Windows shortcuts are a little more all over the place. And I think the placement of the Command key is usually nicer to reach with the thumb than the Control key is with the pinky.

Missing Keys

There are key combinations that replace these keys, but even after over a year, I still prefer the dedicated keys.

  • Delete
    • The Delete key on a Mac is actually the equivalent of backspace in Windows. In order to get a Windows-like delete function, you have to use fn + Delete.
  • Home/End
    • Command + left arrow/Command + right arrow = beginning/end of a line
    • Command + up arrow/Command + down arrow = beginning/end of document
  • Page Up/Page Down
    • fn + up arrow/fn + down arrow
  • Print Screen
    • Print Screen in Windows takes a screenshot. Alt + Print Screen screenshots just the current window. The Snipping Tool lets you point and drag to create a rectangle to copy.
    • You can do all of that in Mac OS too, but I have no ability to remember the proper key combinations to achieve it. It’s like Command + Option + Shift + 4, or something for one of them. I found myself always having to look it up, so I just installed Skitch for taking screenshots instead.

Trackpad

MacBook Pro TrackpadThe trackpad on the MacBook Pro is nothing short of spectacular. It’s incredibly responsive. I’ve not seen a trackpad on a laptop come anywhere close. I had really high hopes for the line of laptops that Vizio came out with last year due to the lack of OS bloatware included and what looked to be a nice design but, despite a supposed focus on the trackpad, it turned out horribly. No thanks.

There are a bunch of different trackpad gestures including different combinations of fingers and direction that do lots of cool things. I don’t take advantage of all of them, but by far my favorite is two finger scrolling vertically in a browser, as well as swiping right/left for forward/back in a browser. It’s truly a pleasure to use in that respect.

Window Management

Size Management

I hate the green button with all my heart.

I hate this green button with all my heart.

The single biggest gripe I have about Mac OS is window management, or the lack thereof. Offender #1 is the green button.

Coming from Windows, your assumption is likely going to be that the Mac buttons map to the equivalent Windows buttons:

  • red = close
  • yellow = minimize
  • green = maximize

But that’s not quite accurate. This is what they really do:

  • red = sort of close
    • The red button will in fact close the current window. The thing it will not do is close the application. In Windows, once you close the application’s last window, the application itself closes. This one is not a big deal really – just a difference. And I close applications in Mac OS quickly and easily with Command + Q.
  • yellow = minimize
    • This is pretty much the equivalent of the Windows function. It minimizes the window to the dock. The one difference is that it takes it out of the rotation if you’re cycling through an app’s windows with Command + ~, which I do all the time, and I find that annoying. I rarely use this though (rarely use it in Windows either).
  • green = do whatever the *%&$ it wants
    • I find this button to be infuriating. Does it maximize the window? Sometimes, but it depends on the application. The rest of the time? It usually makes it bigger at least. I found little consistency, rhyme, nor reason to it. When I asked my boss what the heck it was supposed to do, he said it was the “right size” button. Right size my… After reading up about it, there is apparently some logic to it – it tries to use as much space as necessary in the apps where it doesn’t maximize. Unfortunately, I don’t care for that functionality. I just want the window to take up all of the space on my screen unless I tell it otherwise. Period. I like to take full advantage of the screen real estate and not have random other windows peeking out from behind. I found out recently that, as of Mac OS Lion, Shift + green button will maximize many windows that don’t otherwise. It’s better than before, but still not universal. Extremely annoying.

Position Management

The Snap feature for window management added in Windows 7 is a tear-inducing thing of beauty. If you do not know about it or simply don’t use it, stop what you’re doing now: learn and love.

Windows Snap - side by sideIt allows you to move around and position your windows at will, using either keyboard shortcuts or by dragging windows with the mouse. Great for putting windows side by side and more. Some examples:

  • Maximize window
    • Windows key + up arrow
    • Drag window to top of screen
  • Snap to left/right half of screen (or move to monitor on left/right if already snapped that direction)
    • Windows key + left/right arrow
    • Drag window to left/right edge

Mac OS simply has nothing built-in to compare to this that I am aware of, and it’s a HUGE usability oversight in my opinion. Windows easily trumps Mac OS in the window management department.

But There Is Hope!

BetterTouchToolEnter BetterTouchTool. I recently discovered this free utility for Mac OS and it is fan-tastic.

BetterTouchTool allows you to create your own keyboard shortcuts to trigger pre-defined actions to do things like….window snapping! It also supports snapping by dragging like Windows and is highly customizable. I’ve created the following shortcuts and am in heaven:

  • control + up arrow = maximize
  • control + left/right arrow = snap left/right
  • control + down = move to other monitor

I am so thankful for BetterTouchTool that I used at least two exclamation points above, and I’m fairly anti-exclamation point! (there it goes again)

Posted in pc vs. mac, technical

what if

what if boyMy son is four and a half years old. He is weird and crazy and funny and awesome and awful. It’s great. He’s also the only other man in the house, what with his three older sisters (and his mother), and I have to admit that it’s good to have him around to balance the force a bit around here. I’m definitely not opposed to the increased amount of wrestling and video game playing (except when I lose).

He goes through phases, as kids (and adults for that matter) do. One of those recent phases, which lasted for a few months, was his “what if” phase. His “what if” questions would often come fast and furiously, sometimes seemingly out of nowhere, sometimes triggered by his environment. They were usually hilarious, sometimes pretty weird, and always entertaining. After a a while I decided that I needed to start documenting them, so here’s a sampling.

The “what if” questions listed below were almost always followed up by a statement of “That would be awesome, huh?” (alternate follow-ups in parens)

  • What if
    • our swimsuits had Batman capes?
    • we could jump way up there?
    • this lid could stay floating like this?
    • we could fly?
    • they put our food in a cup?
    • my hand could stretch?
    • Hulk smashed our house?
    • Spider-man webbed our house?
    • it were raining knives?
    • we could turn big in all of our games?
    • we didn’t have any blood or bones? Then we could stretch way out.
    • ninjas’ swords had webs in them?
    • I cried and the drips were blood? (That would not be awesome.)
    • everything could talk? The cups and the bowls? (That would be funny.)
    • everyone looked like this piece of pop tart? (That would be funny.)

I wish I would have written even more of them down.

I really miss the “what if” phase. I think that I need a little more “what if” in my life, both personal and professional, with the accompanying “That would be awesome, huh?” expectation for the results.

Posted in non-technical

automatic image optimization for wordpress

A while ago I wrote about optimizing the images on your website. In that post I mentioned that one of the tools that I’ve used for image optimization is an online tool offered by Yahoo called Smush.it. Well I have an RSS subscription to the WordPress subreddit and came across a post the other day for a WordPress plugin called WP Smush.it. I’ve tested it out and it’s very handy. It uses Smush.it’s API to:

  • auto-optimize of all of the images you upload to the Media Library
  • bulk optimize your existing images
  • optimize existing individual images

There’s also a NextGEN Gallery integration version for your NextGEN Gallery images. I use NextGEN on a handful of sites but haven’t tried out the Smush.it plugin for it yet.

how to use WP Smush.it

  1. install and enable the plugin
  2. smush an individual image
    • WP Smush.it Individual Image
  3. bulk smush
    • pre: Pre-Smush
    • post: post-smush

It even provides you with the shrinkage statistics on all of your smushed images in the post-bulk smush report as well as on each individual image in the library.

caveats

  • the bulk smush is still considered experimental
  • Smush.it has a file size limitation of 1 MB for the original file
  • “Sometimes the Smush.it service goes down or is under heavy load. If the plugin has difficulty connecting to Smush.it then automatically smushing is temporarily disabled (currently for 6 hours). You can always re-enable it via the Media > Settings screen or manually smush the image from the Media Library.”
Posted in technical, web development, wordpress

optimize the images on your website

One quick and easy way to eek some extra performance out of your website is to optimize your site’s images. The idea is to retain image quality but reduce image size by stripping out unnecessary data from the images, making your images smaller and faster-loading.

Image Optimization Utilities

The main tools I’ve used in the past to do that have been:

  • Command Line
    • pngcrush – open source command line PNG optimizer. Capable of attempting various compression levels/filter methods automatically.
    • PNGOUT – Converts GIF, BMP, JPG to PNG (and optimizes PNGs). Claims to produce even smaller files than pngcrush but does not automate the testing of different algorithms/tools.
  • GUI
    • PNGGauntlet – For Windows. Drag and drop UI, capable of converting multiple images simultaneously. Converts JPG, GIF, TIFF, and BMP to PNG.
    • ImageOptim – For Mac. Another with a drag and drop UI and multiple image conversion. Handles JPG, PNG, and GIFs. Uses various different tools behind the scenes (PNGOUTAdvPNG, Pngcrush, extended OptiPNGJpegOptim, jpegrescan, jpegtran, and Gifsicle), trying different options to produce optimal results.
    • Photoshop – Photoshop’s “Save for Web & Devices” option produces some pretty decent results.
  • Web
    • smush.it – online image optimizer (JPG, PNG, GIF) from Yahoo. Also uses various different optimization tools behind the scenes.

Blog Image Testing

I recently took development ownership of a blog site that’s been around for a little while. The owner was complaining about the site having performance issues, so I started taking a look at things.

The first issue was the web host – it’s hosted using GoDaddy’s shared hosting service. Shared hosting (your account resides on the same hardware as who knows how many other accounts, sharing the same resources), although dirt cheap these days, is a roll of the dice to begin with, and GoDaddy doesn’t have a great reputation in that area. According to DomainTools’ reverse IP lookup, there are 6880 domains on the same IP address, which seems pretty excessive. We’re considering moving the site to a different hosting company, although sticking with shared hosting unless the site’s traffic gets too crazy.

The next thing I looked at was a site analysis using another Yahoo tool, YSlow, a browser plugin which suggests ways to improve a site’s performance based a different rulesets. Some of its suggestions are easy to implement – enabling gzip compression, minifying JavaScript and CSS, etc. Some are not – using a CDN, making fewer HTTP requests (especially with WordPress – the more plugins, the more requests), etc.

Properly-Sized Images

The analysis recognized that the bulk of the data being transferred was in images. As a blog site, there were a lot of additional images that have been uploaded by blog authors. I speculated that bloggers may not be uploading optimized images and that this could also be dragging the site’s performance down. Fortunately, violations of YSlow’s “Do not scale images in HTML” rule (i.e., do not upload huge images then scale the images down to smaller resolutions using HTML attributes) were not the problem. Bloggers were usually resizing their images before uploading them, instead of uploading crazy large resolution images.

In the past, when I’ve wanted to do a batch resizing of images for upload to a site (e.g., a picture gallery), I’ve used Photoshop to handle it. Photoshop’s ability to record and play back actions lends itself perfectly to batch resizing images and saving them for the web. It also includes a built-in Image Processor script that can be useful.

Theme Image Optimization

I decided to run an optimization on the site theme’s image files (files used to display the overall UI of the site) first. Here were the results using PNGGauntlet:

  • Original size: 102 KB
  • New size: 36 KB
  • Savings: 65%

Uploaded Content Optimization

Next, I wanted to analyze all of the blogger-uploaded content. I made a local copy of all of the blogger-uploaded images and decided to create some Windows batch scripts to optimize the PNG and JPG files. These were the scripts I used – one for PNGs using pngcrush and one for JPGs using jpegtran. The batch scripts recursively search the main image directory, executing the optimization utility against all files of the desired filetype, generating an optimized version, deleting the original, then renaming the optimized version to the original filename:

Batch JPG Optimization Using jpegtran

pushd "C:\Users\noxad\Documents\src\blog\images"
for /r %%x in (*.jpg) do (
	"C:\utils\jpegtran.exe" -copy none -optimize "%%x" "%%x".opt
	del "%%x"
	move "%%x.opt" "%%x"
	)	
popd

Batch PNG Optimization Using pngcrush

pushd "C:\Users\noxad\Documents\src\blog\images"
for /r %%x in (*.png) do (
	"C:\utils\pngcrush.exe" -rem allb -brute -reduce "%%x" "%%x".opt
	del "%%x"
	move "%%x.opt" "%%x"
	)	
popd

Results

  • Original size: 483 MB
  • New size: 465: MB
  • Savings: ~4%

In this case, image optimization of the uploaded content didn’t buy us a whole lot. A 4% reduction is something, but not nearly as significant savings-wise as I’ve seen in the past, which has been more along the lines of 20-30%. Many of the images in WordPress’ uploaded content have been optimized because WordPress generates various sizes of images (thumbnails, etc.). But at least it was a fun little exercise and I’ve now got some handy little batch scripts for future use.

Posted in technical, web development, wordpress