Jan 27 2014

How I used Criticue to quickly iterate on my PaidCodeReview design

I quit my day job at the end of 2013 to focus on launching a profitable start-up. As I was hacking on a small project, I thought to myself, “it’s only been a few days away from a team, but I miss having peer code review.” Lightbulb. Why not create a marketplace to connet coders who want to have their code reviewed, with those who want to review code? I’d tried launching a start-up last year, but made the mistake of blindly building without real customer validation. I was determined not to make the same mistake again.

So, this time, I created a very basic landing page email catcher explaining the product. I also decided to use Criticue, a service that lets you post your web site and get short feedback snippets in exchange for reviewing others or buying credits (for about a buck a piece). If you use my affiliate link you’ll make my day and get a complimentary review. I’ve found Criticue to be invaluable for early feedback on web sites I build and design by myself. This blog post will follow how I used Criticue to evolve my expert code review service from a confusing mess to a clean and decent baseline in just a couple of weeks for less than $30.

Note: evolution may not make sense in Texas.

Day 1, First Iteration

For the very first iteration, I earned two credits for free by reviewing other sites:

“Well to went to the site and it didn’t work so I will base my review on the screen grab. I personally don’t see the point in a site like this, there isn’t a whole lot of information, its essentially a squeeze page and if i’m honest its not the best looking one at that. You have lot of work to do, I would start over personally.” (1-star)
“First of all you should look professional through your website… and you are not, since image + table takes only half of the screen and the rest i sliterally blank. Next you don’t have any testimonials or quality marks, no pricing can be found and your experts are not exposed anywhere. Anyone can call himself an expert, but to convince others you have to have some evidence.” (1-star)

“Start over” — loud and clear in the very first review I received. Rule number one of using Criticue effectively is to have tough skin. This service provides no value if it gets you worked up and defensive. Despite the bad reviews, a brief period of time on the front page of Hacker News yielded about 40 email addresses. After emailing every one of them and the ensuing discussions with some, I thought the idea had enough legs to turn the email catcher into a basic MVP code reviewing flow.

Day 7, Second Iteration

For this round, I bought a 15-pack of reviews for $19 and used 11 of them. This is what I count as a minimum viable product, in that if you came here you could request and receive a code review. It’s functional, but very rough around the edges, as Criticue quickly points out.

It’s funny, looking back on all these reviews, how much harsher/more critical they seemed the first time through. Look how useful they are! In aggregate, a few things very clearly jump out:

  • the page design is boring
  • concerns about confidentiality
  • add testimonials
  • add examples
  • no clear pricing information
“i am not a developer. trust level right now is a bit low. a proper logo or better typography would help. profile snapshots of your experts would help where you highlight a random person on the landing page to read more about. i can’t load your site by the way. gave me an application error. from the screen shot, your 3 point instructions seem a bit complex. looks like i have to navigate away from this sign up page to create a review folder first, then come back with the URL to sign up. any way this can all be integrated into 1 flow?” (2-stars)
“The defauly bootstrap design mixed with the topic makes me concerned about signing up. My first feeling was ‘is this site just trying to steal everybodies code’. That might not be your intention but there is little trust on the home page. no reviews, who’s using it, or testimonials. The design/information just leaves me very skeptical to use this. It’s not that the site is bad, it just needs more things that build trust.” (2-stars)

I was concerned about using CodeRemarks for the reviews, but it seemed like a decent pre-built solution that would work for an MVP. A recurring theme is that Bootstrap indicates amateur and unprofessional. I don’t know why though; trust me folks, me designing and building a web site without Bootstrap is like going to town with an ugly stick in CSS mode. So I hear you, but really: it’s better if I use Bootstrap right now. The second rule of Critcue is that the reviewer doesn’t have 100% context and you shouldn’t always listen.

“Yes possibly. I’m not much of a developer but you definitely could do with some examples of other work and how using your service will help me with my code. It’s good that I can get started for free, but I’d need convincing before paying you to review.” (4-stars)
“Looks great service. I guess you’l need to provide testimonials, screenshots of reviewed code. Maybe adding a geeky character with big glasses wearing a suit will help to attract+ you can place the reviewers images with short bio, like 10 years of development experience.. this will lead to more trust and confidence and will add value to your services. About your question maybe i’ll try it. But to make developers want to get reviews, they should believe that they will get better after receiving your reviews. I would conclude this by advising you to give the website more graphics, screenshots, colors..etc. Thanks!” (3-stars)
“Good: Great value proposition / USP. Very simple and to the point. Bad: Use some examples below. Number of reviews. How much does it cost. See blog to make the signup process more appealing -> structure: Value Proposition > Key benefits > Call-to-action. Encapsulate. Make a bit better design. Great start to build upon.” (4-stars)
“I would potentially use this on a high stakes project. Maybe alleviate concerns about confidentiality somehow? That and testimonials from past customers would probably be helpful in assuring potential customers. First review free is an interesting idea, but I think you could even get rid of this with some killer testimonials. That and maybe look into updating your design a bit to get away from the classic “Bootstrap-y” look.”

Every now and again, a comment so good comes along that you do need to implement verbatim:

“add more graphics , images and make an expert who is really like a pro. say doctor” (3-stars)

That’s Dr. Paid Codereview, to you.

Day 9, Third Iteration

For this round, I bought a 15-pack of reviews for $19 and used 11 of them. Because so much of the prior feedback said the page was boring and plain, this iteration sports a sexy new syntax-highlighted Vim background:

“I like the idea of what you are doing here but it comes off muddled. The code behind the text you are supposed to read really makes it tough to read the rest of the site. Also the menu items at the top are way too easy to miss. The menu items feel like they are missing. Meet the reviewers and maybe some testimonials should be items to choose from at least.” (3-stars)
“I understand your site is about looking at code but the code background on the site is really ugly and i would just run away from that site if i needed the services you offer.” (1-star)

… at least I tried. This version also added a quick “Meet Our Reviewers” section with bios to lend credibility.

“Seems like a cool idea… Site looks a little “thrown together” from a design standpoint. Confused as to why I need to go to CodeRemarks then come back to your site. What is the end result? Will you fix an error I’m getting? What about an NDA? Finally, how much does it actually cost after your first free review? What’s the turnaround time? The website doesn’t have enough information for me to be comfortable giving you code.” (3-stars)
“Short and simple but if it was me I would want some references on there from prior users attesting to what you have done for them. Additionally atleast for myself knowing abs trusting the people I give my codes to is of the utmost importance so trust has to somehow be conveyed.” (4-stars)
“I like the simplicity of the site, nice! I’m not a coder by any means, but have many years of HTML/Javascript experience. In my opinion, from an end users perspective, I think the best thing you can do is put testimonials on the home page…as close to the top, as possible! This will let potential people know that they aren’t the first to use your service and that others have found your service useful, cost effective, relieving, etc. Just think about the emotions that people go through with coding issues and talk about how you are going to relieve those issues/problems.. just my two cents - cool concept!” (4-stars)
“Wow, simple and to the point. That is one very beautiful call to action. I don’t think the problem is hesitation of your experts more like privacy issues. Like what are you going to do with my e-mail address. I think you just need to tell them that their e-mail is between just you and them. Example: Email Address(We promise your e-mail is between just you and us.) Other then that really cool website.” (4-stars)
“Excellent :-) No fuss, clear message, call to action. Should be right for the the target group (developers). Maybe some small button etc. to get to more information, regarding procedures, prices, etc. ” (5-stars)

And every now and again, you get a review that lets you know you’re really on the right track:

“Holy poop on a stick, they pay people to do code review? sign me up” (5-stars)

Day 22, Fourth Iteration

For this round, I bought a 15-pack of reviews for $19 and used 13 of them. I incorporated a lot of easy feedback. I added a stock image and made the main call-to-action much clearer. The benefits are now very clearly listed. Experts has been condensed, and because the first review took place, I can now include a testimonial from a happy customer.

Once again, a few common themes:

  • needs a real logo
  • more testimonials
  • even more reasons to trust the experts
  • example code reviews
“As I developer, i would hesitate. First thing: how do I know that your experts would help me? I don’t want to spend hours checking if they are really trustworthy. You should expose their accomplishments more. Another thing: how do I know if the expert’s help is helpful for me? Maybe try adding some free trial, so that expert could review only small chunk of my code for free. Another thing: think about exposing some tips & tricks, best practices in form of blog or something similar, this could engage more developers to pay you.” (2-stars)
“I am not a developer but I immediately understood what you had to offer someone who would like their code to be more functional and have less bugs. If I was a developer who wanted this service I would want evidence of the skills that the experts had in this field but I am not sure how this could be achieved. I found that the top of the home page definitely let me know what service you offered and also gave me a quick way to access the service if I wanted to.” (3-stars)
“Just a detail first, the images of the “How will code review help?” section are not aligned. The testimonial part is not explained enough. I would recommend to put an exeple of the case studies you help to solve with this guy. I would like to have more testimonials, one is not enough to win my trust. Maybe a logo instead of a text.” (3-stars)
“need logos and 1 line testimonials up top and clearly displayed. any good press? use their logo logos stamp seal of approval and grab attention right a way. show stock images of smiling clients. or post resume of your “experts” use different slogan other that infomercial “100% money back guarantee”. Make up your own. like “our geeks got your back” or “Sales it’s the bottom line” then use a by line stating a guarantee, and how it works.” (3-stars)
“Really like the site - Great Idea AND look Developers: would you use a service like this? My developer said YES! He would Everyone: how can we remove hesitation that our experts really are experts? - Add a easy to view portfolio of the experts achievements, with a laymans explanation about why each piece contributes to “expert-ness”” (4-stars)

And (for real this time) a few outstanding comments:

“I’m a developer, but I don’t program in Ruby. I’ve thought about doing something like this for JavaScript (I hoped I’d get it by open sourcing some code, but it is hard to get contributors.) One way you can show that your experts are really experts is by making them contributors or better yet patch approvers for open source Ruby code that has a good reputation. Ideally it would be a reputation for having a good codebase, although your customers might not be aware of that. (I’m thinking in terms of Raymond Hettinger doing python consulting.)” (5-stars)
“I’m a business owner and I am wondering how much the bad things you mentioned would impact me. So the first thing I would say after visiting your live site is, tell me why I need this. What does bad code do? How can that effect the way my site runs? Do I lose PR or fall in the SERPs? What does Google think about bad code? As a business owner I want to be found and I want the visitors to my site to convert into customers. Does bad code make my page load slowly? Does it make it difficult for browsers to read? Tell me how this helps me make money and convert visitors into customers. :-)”

I’ve been happy paying $1 for short and simple reviews, but this last review in particular is so much more valuable! This is a potential customer telling me in great detail that she doesn’t understand how this thing I’m offering will improve her business.


In total, I used about $40 worth of reviews (I also used the 15-packs on other projects) and a handful of free ones. I recommend buying a 15-pack if you’ve hit a milestone, otherwise idly reviewing is good for just a few reviews per week. Pro tip: two 15-packs for $38 is cheaper than one 30-pack at $39

An added benefit of Criticue is that the users of the service are generally in a similar boat - the early stages of building a web service. You can respond to reviewers who like your product and you might even find your first few customers this way.

Overall, Criticue is well worth ~$1/review (or free if you’re on a budget) both in aggregate and for the top 1-2 insightful reviews. Criticue has changed how I build web sites from the ground up for the better.

Next steps

I’m a programmer and it’s been a constant battle to not dive in and make the web site perfect (and then make it perfect next week, and again next week, ad nauseum). Every single person who’s been advising me has told me to STOP programming and get more customers, though, and they’re right. This design is good enough to shop around to potential customers, and I can fix small things that don’t cost a lot of time (for example, I’ll replace the money back guarantee tagline as soon as I can think of a better one).

I’m currently talking to people who have paid contractors to review their codebases, trying to find communities for people who might manage outsourced devs (bootstrapping meetups, for example), writing blog posts about the benefits of code review, and asking people with large followings to give a shout-out to their fanbase. I think there are definite advantages to programmers using their service personally, but I think it’s going to be harder to get them to open their wallets at first.

If you’re reading this far because you wanted to know why I just happen to have screenshots from each iteration of my site, it’s because I use DailySiteSnap (another project of mine) to automatically archive a screenshot of the homepage once per day.
Comments (View)
Dec 02 2013

How I started a scholarship in my hometown

About one year ago, I wrote a blog post announcing the Derewecki Science Grant. It’s a scholarship I started for high school students in my home county of Coshocton, Ohio, in the form of a one-time $1000 award. Over the past year, many people have asked how one goes about setting up a scholarship. It’s a lot easier than you might think, costs less than you might think, and can be a rewarding way to give back to the community that brought you up.


This is going to be a little blunt. I was a smart-ass in high school. I knew how to computer, learned how to teach myself things on the internet — I was the smartest person I knew. I mouthed off to teachers who I didn’t think were doing a good job. As you might expect, when it came to subjective awards based on votes from a committee of teachers, I drew the short straw. What grinded my gears the most was becoming co-valedictorian after being rejected twice from National Honor Society.

I’m the first to admit that I was often a smart-ass, but I still think I was treated unfairly by more than a fair share of teachers. One of the teachers on the NHS committee that had denied me membership actually held me after class to apologize — he was one of the good ones. A few friends and I dreamed of a future where we could start a scholarship that could only be given to non-NHS students, a scholarship so tantalizing that it would demolish the relative appeal of the NHS.

A bit petty, but I was 17 or 18 and I like to think I’ve grown a lot since then. One thing that hasn’t changed is my belief that some high potential students in my hometown are overlooked for issues that are easily correctable. In my case, Coshocton just wasn’t my trip — I didn’t fit in. As soon as I got to college, I ate humble pie until I puked and adjusted my attitude on life. I was lucky enough to be able to attend a college with a generous financial aid package, in addition to the merit-based scholarships I received. But I had a friend who literally didn’t have the money to apply to college or anyone (his own father included) willing to co-sign on a loan for him. So he didn’t. This was the only friend I had in high school who was into programming, and was the only I saw regularly who pushed me further. And it’s totally unfair that he didn’t get the chance to go to college. So I decided the best thing I could do was start a scholarship for people like him, people who could’ve easily been me.


Deciding that you want to do it is half the battle. The other half involves actually setting up the scholarship. I sort of took the easy way out, but I think most of you probably can too (unless you have complicated requirements). In my hometown, the Coshocton Foundation gave the majority of local scholarships to high schoolers, so I just reached out to the head of the foundation with my proposed selection criteria. The mininum amount is going to be up to the foundation you’re working with, but expect it to be on the order of $10k. Do the math, figure out how much you think you’ll be able to put in and how long you want the scholarship to last. After a few back and forth e-mails, we hammered out the details and out came an official looking document:

I happened to be home in late March, and I met with a small committee at the Foundation to go through all the applications and vote on a recipient. We gave each applicant a fair shake, and one was a clear standout, so the decision was unanimous. It should be noted that because of the Pension Protection Act of 2006, this is a committee in which I cannot have a majority vote. This means that the winning applicant might not be the person I chose, but this seems fair in exchange for the convenience of having the whole thing basically managed for me.

Online Fundraising

At the time I set up this scholarship, I was working at so it seemed natural to launch a campaign to raise money online:

Give to the Derewecki Science Grant
For my birthdays and holidays, I’ve been asking friends and family to give to my scholarship, and have raised $585 to date.

Company Matching

If you or your friends work at companies that match contributions to charity, that’s a pretty easy way to double the impact. When you find out which companies match, actively recruit your friends who work there. This tactic raised an additional $3000.

Meaningful Giving

Overall, it’s been a very positive experience. Unlike other philanthropy I’ve done, it’s not an anonymous impact. I remain connected with kids in my hometown who can surprise me in all sorts of ways.

I originally had a coding challenge that was part of the requirement, but at the urging of the Coshocton Foundation, I changed it to something more general:

Summary of something applicant has built or created: for instance, a piece of sample software code, a woodworking project, art project, “souped-up” hotrod, home-built segway, robot that rakes the leaves, etc, The goal of this requirement is to provide an opportunity to “show off” something about which the applicant is passionate.
In retrospect, my original criteria was way too restrictive and I think I would have been disappointed with the applications. By broadening it, a student was able to sell himself on the hotrods he had been building with his dad since he was 8 years old and going to school to be a mechanical engineer, and is the first in his family to go to college. After just one year of giving, it’s already been more meaningful philanthropy than anything else I’ve done, and I look forward to being surprised and delighted every year for a long time.

Related Posts: Announcing the Derewecki Science Grant

Comments (View)
May 24 2013

PayPal’s insidious inverted unsubscribe page

This morning, I clicked the unsubscribe link from a PayPal marketing email and was taken to:

Notice anything special? While these forms usually tell you to uncheck the box if you no longer wish to receive emails, this one makes you check all the types of email you wish to not receive. I’m sure this has had a measurable impact on their “unsubscribe conversion.” Well played, PayPal.

Comments (View)
Page 1 of 25