MPMoriarty and I have been working on a new scoring system that we think will work a bit better. Rather than a simple scale of 1 to 10, we’ve broken things down into sections so an OS could get high marks for usability, for instance, while low marks for limited options and customization (or vice versa, or whatever). Check it out:
Functionality
How powerful is the operating system’s feature set?
[4] Full: Full support for basic and advanced features
[3] Partial: Full support for basic features; partial support for advanced features
[2] Basic: Full support for basic features; no support for advanced features
[1] Poor: Poor support for basic features
[0] None: No support for basic features
Ease of Use
How easy is it for the average user to work with these features?
[4] Excellent: Implemented very well; little room for improvement
[3] Good: Implemented well, but could use minor improvements
[2] Average: Usable, but could use minor improvements
[1] Poor: Usable, but needs major improvements
[0] Bad: Unusable
Availability
Are the operating system’s features included in the base installation?
[2] Built in: All features are included with the operating system
[1] Separate download: Requires a separate download for some features
[0] Not supported: Not included with the operating system, and not available as a separate download from the manufacturer
And here are some sample sections, shown how they would be scored with the new system as compared to the current one:
• Applications > Web Browsing
Mac OS X: 9 (currently 8)
Functionality: 3
Ease of Use: 4
Availability: 2
Windows XP: 7 (currently 6 (7))
Functionality: 3
Ease of Use: 3
Availability: 1
• Applications > Video Editing
Mac OS X: 0 (currently 1)
Functionality: 0
Ease of Use: 0
Availability: 0
Windows XP: 7 (currently 7 (8))
Functionality: 3
Ease of Use: 3
Availability: 1
• Files & Folders > Sorting & Grouping Files
Mac OS X: 6 (currently 5)
Functionality: 1
Ease of Use: 3
Availability: 2
Windows XP: 9 (currently 9)
Functionality: 4
Ease of Use: 3
Availability: 2
Note that this also simplifies the scoring system by removing the “adjusted” scores. Instead, we’ve kind of flipped how this works - any features that are available through a separate download are just included in the regular scoring, and then the OS is knocked a point for not having the feature built in.
Can’t say I agree with that scoring. All it’s going to do is erase the tie that currently exists on the site because XP will get a point less for each free download.
Linux will also get heavily penalized when you add it to the site as well.
<not related>Why does XP get a 1 for availabilty in Video Editing? WMM2 comes on all new Windows XP SP2 discs.</not related>
It also makes things complicated because you guys are going to have to define basic and advanced functionality for every single topic.
I do like the fact that this makes the scale truely 0-10 instead of 1-9 as is now.
Adding and ease of use score is helpful to readers but penalizing for downloads is not. You already separate the scores to let people know what is a download and what is not so why are you taking further action?
IMO you guys should be using your limited time to prepare for Vista. You should even add it to the site. I know a ton of your readers what to know how Vista stacks up to XP and Tiger and they want to know that information now, not on Vista’s launch day. Most of your readers are going to be making their purchasing decisions between now and launch not just on launch day.
Now that Vista is freely available it make sense to add it to the site for at least academic purposes. Re-scoring the content you already have with another equally flawed method is useless IMO. The scoring method needs to be designed around specific types of users or uses and, most importantly, weighted in some way.
A section like Fonts shouldn’t get more weight than Finder vs Explorer or Web browsing. If a section has multiple sub-sections then you should give an average score for the section and use that average for the final total. That way you can feel free to add as many sub-sections as you like (to help differenciate between similar features) without overweighting any particular section in terms of the final score.
Most of it is ok, but the following section is really bad and will cause nothing but trouble
Ease of Use
How easy is it for the average user to work with these features?
There’s a few problems with this:
1) How do you judge it? Whatver you decide on, it isn’t based on anything but your own opnion. How do you say judge iTunes? Using it simply the ease of use is very high. Start digging a little and it gets a hell of a lot more complicated and ease of use strats dropping quickly
2) What’s “average” ? How can you score a section on what the average user would work with these feature, if an average user would never touch that feature? Does an average xp user knwo to always use both buttons and that just about anything on the screen has propetie sof soem ort accesed by a right click? Does an average mac user know by heart numerous modifier keys? If an average user doesn’t even know or use feature X, how could you possibly score that feature?
Of course we are going to have to make judgement calls with some of this, but we’d have to no matter what the scoring system is. And in fact, we’re making judgement calls with the scoring system we have NOW, only its difficult to see the reasoning behind it some times. The way that this new system is broken down is basically how we’re scoring things now anyways; the difference is that instead of using a flat scale of 1 to 10, we’re showing you where each OS excels and where it does poorly. So it’s not so much a change in how we’re scoring as a change in how we’re displaying the scores. That’s why in my above examples, you’ll notice that the scores stay largely the same.
Also, this system would give us a chance to compile some interesting final scores in addition to the regular total number of points. For example, we could add up just the functionality points, or just the usability points, etc. to create subtotals showing which OS is stronger in those specific areas.
There’s going to be subjectivity to this no matter what you do.
In my opinion, this scoring system benefits a company that throws every single feature under the sun into their OS but implements it very poorly. You automatically get points for functionality and availability no matter how easy it is to use with no criteria about whether or not the feature is even needed.
There’s going to be subjectivity to this no matter what you do.
In my opinion, this scoring system benefits a company that throws every single feature under the sun into their OS but implements it very poorly. You automatically get points for functionality and availability no matter how easy it is to use with no criteria about whether or not the feature is even needed.
It equally favors a company that implements few features, but an insanely usable UI for those few functions.
Can’t say I agree with that scoring. All it’s going to do is erase the tie that currently exists on the site because XP will get a point less for each free download.
Is that so wrong? I don’t know about you, but I find that having something included OOTB is preferable having to hunt it down on Microsoft’s site.
Linux will also get heavily penalized when you add it to the site as well.
I disagree. It locks the minimum score for all sections at 4, for Linux (whenever it gets included). Before there were many sections that would have scored a one or two (Audio/Video conferencing, for example). None of the sections that would have scored highly for Linux would be affected by this.
It also makes things complicated because you guys are going to have to define basic and advanced functionality for every single topic.
They were effectively doing it already, I think.
Adding and ease of use score is helpful to readers but penalizing for downloads is not. You already separate the scores to let people know what is a download and what is not so why are you taking further action?
I think they would need to stop seperating the scores at that point. Distribution is a valid point of comparison, however. I personally know many users who don’t know of many of Microsoft’s free downloads. To them, those features don’t exist. It would be one thing if Microsoft provided an interface in, say, Add/Remove programs where a user could install all of these freebies with one click. But they don’t do that, so there *should* be a penalty.
IMO you guys should be using your limited time to prepare for Vista. You should even add it to the site. I know a ton of your readers what to know how Vista stacks up to XP and Tiger and they want to know that information now, not on Vista’s launch day. Most of your readers are going to be making their purchasing decisions between now and launch not just on launch day.
I think most of XvsXP’s readers have already made up their minds about it.
Now that Vista is freely available it make sense to add it to the site for at least academic purposes. Re-scoring the content you already have with another equally flawed method is useless IMO. The scoring method needs to be designed around specific types of users or uses and, most importantly, weighted in some way.
The scoring system needs readjustment, badly. Otherwise Vista is going to have to dump a bunch of 10’s in the Windows column, despite clearly leaving room for improvement.
I think this is a great system. The two main points to any piece of software are what features/functionality is has, but also how easy it is to use them. This benefits both OSs, as Windows generally wins in places in the functionality, just for the amount of micromanagement you can do with the system, but OS X often wins in the ease of use. It also eliminates the whole two scores for downloads bit, because it gives the weight to the actual feature, regardless of it being a download or not. Tiger will beat XP here, but remember, Vista is round the corner, and it will have a lot more bundled
And speaking of bundled, I think there is a chance here to allow in typical bundles with the computer. The availability of these would normally be 0, right? They are not free, however they are included in the price. But when an upgrade comes out you have to pay for it. Surely the possibility of a -1 availability score would be usable, allowing for things such as iLife, Works etc to be included in the shoot out. It would offer a more comprehensive look while allowing it to be fair.
There’s going to be subjectivity to this no matter what you do.
In my opinion, this scoring system benefits a company that throws every single feature under the sun into their OS but implements it very poorly. You automatically get points for functionality and availability no matter how easy it is to use with no criteria about whether or not the feature is even needed.
I assume you’re talking about Windows here. So lets say in a certain section, Windows has great functionality (4 points for supporting all basic and advanced features), and it includes this functionality with the OS (2 points). But the UI is terrible, to the point that it makes these features unusable (0 points). It gets a total of 6 for that section.
Now let’s look at the reverse. Let’s say Mac OS X has a fairly decent UI for something (3) that’s included in the OS (2), but is very limited in its features (1). It also gets 6 points.
So if your argument is that Windows would get an unfair advantage because it has more features than OS X - despite the UI not being as good - that doesn’t fly. Functionality and ease of use both operate on a scale of 0 - 4 points each. So in terms of scoring, great functionality with a bad UI is “equivalent” to poor functionality with a good UI.
Of course we are going to have to make judgement calls with some of this, but we’d have to no matter what the scoring system is. And in fact, we’re making judgement calls with the scoring system we have NOW, only its difficult to see the reasoning behind it some times. The way that this new system is broken down is basically how we’re scoring things now anyways; the difference is that instead of using a flat scale of 1 to 10, we’re showing you where each OS excels and where it does poorly. So it’s not so much a change in how we’re scoring as a change in how we’re displaying the scores. That’s why in my above examples, you’ll notice that the scores stay largely the same.
Form xvsxp’s introduction page
his website attempts to filter out all of this white noise by providing you with exhaustive comparisons between Mac OS X and Windows XP that are fair, objective, ubiased, and — most importantly — based around facts rather than opinions.
Can you explain to me right now what an average user is? I want a strict definition of what they are and what they are able and not able to do. So far not a single person has ever been able to define that on xvsxp;. Sometimes it’s someone with the inability to move a mouse above a program and get a menu, but in the next breath it’s someone who understands what a linker and compiler even is. Just saying average isn’t going to cut it
Now, give me concrete evidence that program Y is more usable then program X. How do you prove that? By asking people’s opinions and basing the score on that? by basing it on your own opinion? How does pure opinion suddenly count as score?
Do you see my point? You are not basing the usability score on facts, because by it’s very nature it doesn’t have facts, only opinions. OSX could get a 1 for general; usablitly because it’s menu bar is horrible. Worst GUI invention ever. How do you prove that right or wrong? What evidence is there, based on just facts and not any opinions, of how usable it really is?
I think this is a great system. The two main points to any piece of software are what features/functionality is has, but also how easy it is to use them. This benefits both OSs, as Windows generally wins in places in the functionality, just for the amount of micromanagement you can do with the system, but OS X often wins in the ease of use. It also eliminates the whole two scores for downloads bit, because it gives the weight to the actual feature, regardless of it being a download or not. Tiger will beat XP here, but remember, Vista is round the corner, and it will have a lot more bundled
Precisely. Each OS gets scored appropriately for its strengths and weaknesses, rather than just mushing that all into one and coming up with a flat score out of 10. Also, you’re exactly right on the downloads. Instead of giving points for functionality added through a download (which is what we do now), we score the download’s features just as if they were part of the regular feature set, and then ding the OS one point for not having those features built in (since a download ALWAYS requires some extra effort to get and install - AND you have to know it exists in the first place!). Thus, downloads are still taken into account, but we don’t have to have separate scores. This will eliminate scenarios where, for example, XP has 4 different scores for one section (Pro, Pro + downloads, Home, Home + downloads).
And speaking of bundled, I think there is a chance here to allow in typical bundles with the computer. The availability of these would normally be 0, right? They are not free, however they are included in the price. But when an upgrade comes out you have to pay for it. Surely the possibility of a -1 availability score would be usable, allowing for things such as iLife, Works etc to be included in the shoot out. It would offer a more comprehensive look while allowing it to be fair.
We’re staying away from iLife and the like, at least for right now while we’re working on this new system. Incorporating that and other bundles would be a big change in the scope, and we really don’t want to touch that.
his website attempts to filter out all of this white noise by providing you with exhaustive comparisons between Mac OS X and Windows XP that are fair, objective, ubiased, and — most importantly — based around facts rather than opinions.
Can you explain to me right now what an average user is? I want a strict definition of what they are and what they are able and not able to do. So far not a single person has ever been able to define that on xvsxp;. Sometimes it’s someone with the inability to move a mouse above a program and get a menu, but in the next breath it’s someone who understands what a linker and compiler even is. Just saying average isn’t going to cut it
Sure. An average user is someone who knows more than just the absolute basics (moving the mouse, for instance) but not enough to mess with the Registry or write code or type long commands into the command prompt. Someone who knows email, web browsing, Office, installing software, working with photos off their digital camera, playing music, etc.
Now, give me concrete evidence that program Y is more usable then program X. How do you prove that? By asking people’s opinions and basing the score on that? by basing it on your own opinion? How does pure opinion suddenly count as score?
Who said pure opinion counts as score? We make the comparisons as factual as we possibly can. No matter what you do, there is ALWAYS an element of personal preference that you can’t quantify. So we try as best as we can to not score based on opinion. In some cases, we HAVE to make a judgement call because there’s simply no way around that. When that happens, some people agree with us and some people don’t. That’s the way the world works. We cannot prove with 100% certainty that program Y is more usable than program X, but we can try our best to approach that. That’s the goal.
Do you see my point? You are not basing the usability score on facts, because by it’s very nature it doesn’t have facts, only opinions. OSX could get a 1 for general; usablitly because it’s menu bar is horrible. Worst GUI invention ever. How do you prove that right or wrong? What evidence is there, based on just facts and not any opinions, of how usable it really is?
Not necessarily. Yes, usability does involve personal preference more so than feature set (since that’s easier to quantify). But to say that there are NO facts involving usability is wrong. Let’s talk about OS X’s menu bar. It is a FACT that you cannot vertically overshoot the menu bar because your cursor hits the top of the screen. It therefore follows that it is a FACT that OS X’s menu bar requires less vertical dexterity to work with than Windows’ menu bar, which is not at the absolute top of the screen. Whether you PREFER one menu bar over the other is your personal opinion. But it is a FACT that in terms of vertical dexterity, OS X’s menu bar requires less, and is therefore more usable within that specific area. So I disagree that there are “only opinions” when it comes to usability.
Do you see my point? You are not basing the usability score on facts, because by it’s very nature it doesn’t have facts, only opinions. OSX could get a 1 for general; usablitly because it’s menu bar is horrible. Worst GUI invention ever. How do you prove that right or wrong? What evidence is there, based on just facts and not any opinions, of how usable it really is?
Functionality is purely opinion as well in that case. You may think that something deserves a 4 while I may think that it deserves a 2, depending on how we feel about it. IMO the OSX menu bar is much better than the windows menu bar as it is never obscured, always in the same place, helps show which application is in the front and allows for applications to be open with no windows, which is EXTREMELY useful.
Any review of software will always have a level of subjectivity no matter what scoring system you use. Hardware can be benchmarked, software is a lot more difficult..
So lets say in a certain section, Windows has great functionality (4 points for supporting all basic and advanced features), and it includes this functionality with the OS (2 points). But the UI is terrible, to the point that it makes these features unusable (0 points). It gets a total of 6 for that section.
Now let’s look at the reverse. Let’s say Mac OS X has a fairly decent UI for something (3) that’s included in the OS (2), but is very limited in its features (1). It also gets 6 points.
So if your argument is that Windows would get an unfair advantage because it has more features than OS X - despite the UI not being as good - that doesn’t fly. Functionality and ease of use both operate on a scale of 0 - 4 points each. So in terms of scoring, great functionality with a bad UI is “equivalent” to poor functionality with a good UI.
You are only comparing one feature. You would then have to add a few more features to Windows that are implemented poorly or features we don’t even use that aren’t in OS X.
It seems it’s easier for a feature to be functional than easy to use, in my opinion. Maybe I’m wrong, but I think you are more likely going to give a poorly implemented feature a 4 for function than you are going to give fewer features in OS X a 4 for ease of use and functionality. Make sense?
In your scenario, you are giving Windows a 0 for ease of use with the file manager. Are you saying you are really going to give Windows a 0 for that?
Sure. An average user is someone who knows more than just the absolute basics (moving the mouse, for instance) but not enough to mess with the Registry or write code or type long commands into the command prompt. Someone who knows email, web browsing, Office, installing software, working with photos off their digital camera, playing music, etc.
Then what is the site about? One section is about newbies, next section is a very advanced user. So on those same sections you re now going to define n average user? There’s going to be times that within a paragraph your definition of who’s looking at this site is going to change. making that score is only going to magnify the problems
Not necessarily. Yes, usability does involve personal preference more so than feature set (since that’s easier to quantify). But to say that there are NO facts involving usability is wrong. Let’s talk about OS X’s menu bar. It is a FACT that you cannot vertically overshoot the menu bar because your cursor hits the top of the screen. It therefore follows that it is a FACT that OS X’s menu bar requires less vertical dexterity to work with than Windows’ menu bar, which is not at the absolute top of the screen. Whether you PREFER one menu bar over the other is your personal opinion. But it is a FACT that in terms of vertical dexterity, OS X’s menu bar requires less, and is therefore more usable within that specific area. So I disagree that there are “only opinions” when it comes to usability.
And then there’s the fact you can’t “loose” a window in xp. You always know what commands are with what window, and it’s a far better visual indicator on windows. The fact you can’t really change the font itself could make the menubar hardly usable Blah blah blah, etc etc etc
The example isn’t important at all, it’s the fact that within 5 minutes i could poke some serious hole sin your reasoning. I know you guys talked about it, but I don’t think you put enough thought into potential problems with it. If I can poke just a few hole sin one single section, imagine what the general forum, can do with a hundred of these sections. How factual are you going to be that many times in a row? how many times is it just going to come down to “we said so” ? isn’t opinion what you vowed not to do because that what was making the site bad before?
Any review of software will always have a level of subjectivity no matter what scoring system you use. Hardware can be benchmarked, software is a lot more difficult..
You are proposing a section that’s mostly based on subjectivity. That’s a major problem
