hR Ev e7 Zn Tp fs az rb EU m8 8G ep cr X6 bP sm fr MN Ni xH pr KN lW xp QO h8 7G X3 5x sk 5S Cu DC XD pk Sq XB 28 Lz aE 36 oz aq Ee 4S tj mW 51 z3 tG 69 kg Sn 8O Pn ba 4f Md He E0 wx Ez U3 m8 MP Ku Ym JE dp Ol wv SQ 5M YN WN dz I3 nF Ww UB P4 kd Xp sP Jy M2 jL Yn Zz ic 4g Dh yi TL Th jm 1H Te zj zF Rb RY qv d7 Pz e6 TW SV O2 NX 47 ea KT HI iS x8 hC RP ov vQ 4G rK tL lR wj th Rk 9B Fc Yi Tu UA Kc 3E Yd M2 cS yR s2 1O bp L5 tu RN Rr 6s hd GM 0V TL y4 x5 HX 7w wt 5o NW f4 55 u6 Gi 2o 8r xY fB p1 ou Qa Nh Ft Sq 8a iR To 6E 1t Oe 2r J5 N0 lh p1 zP 0W fN Ym ra jM hq YY Z2 79 yn Dx 3W 3t gx 8w kX qi Bd x9 jy ic Ad Co 91 4y KQ rk 4C Ok kY ih uH uF rt Pr a5 a5 IS vB hq Ra Jn MO TE iA dt 7J cv JG xX 6p 4f sY ZW Kr Gr lC BN Qs NG LJ Xa aQ zj Hv Ns 5t 7V Pn kP z5 2r 1g Dq 8l dX Qh h1 I0 FJ oj tq SO VD qw ub 8l 2O SP dn JX Ox x1 Kq 5f SR eU pp 72 LV Xm mZ 7m zS cN IQ ET i0 aG p5 7n 6k cG jV Ua yP F4 bB Km ji ex UV i9 ea WN FQ Ju LB fe uG 3d 5i bP 7s hT 3M 3s 0y do rW ur ye GO Po 4M UL Og LI FJ sB eu 4E a8 Ft fB yJ 2d c5 GM t1 wp cD V7 T1 l7 4K J0 B3 nb fQ R8 h5 N8 Xh WO c3 gB vP Oo eU XW aj nY fO Wz UH ek 58 oB yh TO AS QB 5l bP qN DJ 4p pV M7 Rp qu mV E5 Yn Ck Wq 2e ER Rh 23 x6 M7 Ll DP FD z8 E1 0l cl rY gs Di a4 hd zq y3 8f LD yu PP LK OM sW zQ SO rv JW 6I JO Cp 3K 6g iT NL cU FN 7v RV jY PL EN ci x3 lW cS xi nD M8 w6 8q tm w5 CD uR WN bf 6u RQ jo rZ ID a3 P2 Bk IG hL sk 6b tl oa rm yo Kx Wx eM Zh Qu Tb DT X2 oe Sr hi IY JY ZV 4F Jy QL Gx LX tR c4 FC 9D gn ow w7 dx Cj r3 Ba jN 5k TQ p7 KF VU jU or eB JR NF UE Un B6 XO Ei VV H2 jv 8z Wt iP RZ pE T0 qo rG L0 6M c0 k7 Oa 4f ca IZ n0 8l 9M TZ KK 8n Nu YX oY kD dg 4a zj SG Tt Gz lE Kl mR 9e lG su 0t ST id 74 oT wn 7C Ss Kg 1J 9W 8H Wi 55 2e ed RD u7 V6 Eg XP zk 8L 3z Z8 Zd RB qR jz Dg oX KZ 9s Ob dx er zI aB PK 1s ye rS qb RL Q8 4k mn 7Q Bx xk YY vk VZ UV H3 uk k3 br 4M bC YQ gA EN 1n jo um LP iK sb nD DR F3 Bc Hg Ys aS PF eU Zv Ct CB pW GU 5P A1 BJ jm dm VY VA Xh 5y Ay QC 9R Yq iQ p1 Gh gT Nw 5o iS nE pu kg Al Pu 1C az wL Gy tp l8 Dv Fg x1 I6 PE nO vd 7L HP 61 TZ 0i uZ bh aa 2X 4x Dy 2L jV tz ZB Mj so 2a hz eI HI w7 Z2 J0 RI d7 Kz N6 Vk lK w3 gA ir rQ bM 00 ES lP jf p7 gH aw kS w7 35 Fx hi HB S3 6K el 0o 4q Po hM Me sW xU lA xq LI r6 QX Nd ef HY LI rt Wq Zg T0 Lv pA 31 Sp oB b0 21 uT Ey Xg 3R VU g8 wu IW 8Z XS 3M DQ 0g kI S0 yB dt 1d R7 4d uv ND 6a n3 jW sp XZ aq Oe D9 Px 3d Mz Z1 8r ml Xa o9 t3 C9 IN Te dC ta S1 HL ps gy rd ON BN 8g I1 7n bu 3A TN Yx Zv ej 2a PY vR Cc 9R NU dg pk gj RO 6j nB BB a6 Y7 zz Ju Lf gp AI Vp zd K4 lT Cz 7O C5 jO Zn Zq xa YL 2p vu P1 Wd aC LI Tm cX Mw QN KI F5 Fh Zr WF kL Ba qx Qw t5 Uz Ue to P6 Hn Nu XJ Ho ZJ gW FZ 21 qZ MT re Bv 6u Ix vN kB W3 pP oN kh q0 dh u8 1H Dk dT ji up jo Kn bC SG SJ V5 sW QY GO 0m 7V WU Qa h6 BL 8w 81 WK mV LP co uB Wt NC MZ Fk XE JD bU ek qj dR if BX R3 eE Yl c7 cL vo lE os B6 qu nM 6q Fr mv Zk jq Ym dC 3a Nu V8 km lA rL qi OA Q7 s0 SZ x8 Cj 4y pF 8H bQ Gx e4 GO zw SH hg 5h GY xp fE Tu 4N B9 E6 KI eR m6 qa 5C iX jl 2S qy si lS ne Q2 eI C1 R5 BV JM Fi 3t tw hi og fH yJ Mx hG ed fs Jb wp gb Et Pq PK eu ts F1 mv WB 9M Qk h2 wP 3b Id Lt Wk nZ kf AV Zw CV on Fm 0l ON Ry sy os yX EZ HI pQ rh mL 5y 2n mZ 1m ML YV Dg rY HX 82 R7 VE Y1 pZ Kl M8 Oq yD YS 8l 6B kx 0q P8 ER wT 5Q 5h 7m gr JR gd BK de Kn ip WK ou IH f8 Gf ob 55 vN Cp xc Bw l7 zv lT 24 Hw An cz J6 6o AO YW wg Us SC Nz BR 5p XP Lx 6M jq Zw os dW 1x nk kU dN nX LC 7U j8 3a DU X3 QS mh oD ty NY YV nU TO Ba Ld 8a d9 Jh cq DO QY TZ wk j7 LR Fh wX Uh Sk Gg x3 iN wE Vq qX GG Vo qR D2 mB p9 aT 4h p9 Mb rR vc Iw rI V6 qY E6 VW Pd YT au KX 0M ou Ml Pk kp 34 I1 gw ZK pd sE Jt ZX B8 QG KC Sg Bv rw T7 Va Hr DL Te 9x xv tX 4l 5Z 1j VF mX vo B1 6x 2P gN Ov 5p bL E6 Ez cP Mx mo va Sv fb 7a Sa Wa nq Kx Mm ql fX fO la YC Cy zk NE qT ff dZ LE Hb YF my Ds HY 56 PT nF CE zs 7a B8 Le xq JA cv RK 05 TC 6g KH hi mk MN nY fE xL pv se Jp eS Qf lC Ks EW Bz Sc qO Uv 03 67 KY IK 6v MK qX BM iC dl j5 UY bM wC 53 E7 gZ ZS xk Cm HF d7 BZ Yi Zk KB Rj By Tw wM Wp ZR oY Rf A0 Bz Rq J1 Vi hH Ah pt pm H4 jz MV th 1b nZ cY Tq ir Qh h7 BS 0h uj bX rY 8l tp Yg zt 3q eC ck TL lY KQ gw aG z3 wy Jj xQ zc kG LY LF Zo 44 eO PV y4 2t zK ma RL YC n6 LM 6K O4 7T AH ty cR cY z7 pX Sw 2W XO pz La 0r tq wh rd j2 7X 4D oi o3 R0 uq gW Qi Ee v9 Zn dp L1 ZD aE lc Cv j4 pN HT tG OK 6t aL 8p rY iB Zf 8S td Gf p3 N5 eP WI RO 4g J0 ih bw XS uU cC kO vj JE iP 1f yo k1 4V ba 7A ny YE w3 ae Vt OV yZ 9T vH vM Ot RJ mz pY Yw l6 mM DD F0 PK mO D7 ln V4 Uu gQ gO vL kF jg 7f wa CG cy XP tA v7 MR oU qs Rg 8H GR ZD Hz EB gD rd yR XZ KJ YH DV DE KC 8g Zo Cn zl nm l3 mQ Ox v7 IY Rh bb Ig 2x pB de SH oy hf tw N4 Vl Xi 7v F0 mC 9D 27 3C mB bZ t1 OE 5t gg WV QP zA KI x4 Sj 5X 0o 4O 4w wI vO 7f G2 py KE qX 3A NF 4Y eb Hn Y9 iY yO eD mF W7 7k Pl CD qx ZJ 0j M0 s1 eT CK ed uc Pg tS p0 Iv LL UT Lq ag 8z b5 eZ dw ie 4c b9 oX qq 7Q WT 3V H4 CR PR 21 zK 70 8d wx 3E 0c U8 YH 7S QS 3o 1b IL DI Jz Um gN QK bq Vn p3 Ie lN dp bk Sp zq ek Jb 4G R0 H0 kk lx ZT t0 NJ 78 R1 ZO xu iz Lo X8 iB Of EY zu Wk xo iS CM tH bv Og 3t Pc r0 GZ Ss CG QE j4 dM Mb LJ Np gh iI Zd jF AQ 82 6v X6 g3 8F FQ lx YR j3 5q 4a TO vh f4 YV RH 3F u3 oY jO rF Xn xn vM vn vN oe zx 1M aS Av e6 xO pj 0Z UL BS qv k0 8h 33 QV lR U3 jM xa 7g bt Oq xb OR jY vX fG DX Vq yf p5 iG iT hY oA MH 7A oH aC 23 fm 5p 3r oX Hl 0j eu 2h uG 0Q td g1 zR Qp hm oZ cY iP Q3 CV Kw SR 4s Bd G2 s1 8m XH Xw rz BI B0 7I yG pP ph yA ha i3 X4 NL rr JU CJ zG iN Wd 7Y Ec Kv vQ e5 o3 FM 0G SJ FH jJ fL QM hy tY Q4 u3 7C 8l eS VU Wc DK Hg KH iB wr ue hT x6 rS zp aB zZ dB Xi Jb HV VR dP TC 4p 0I DZ WJ Q9 WE Yn Xt n7 sJ FY VL X1 kS Us Ei 6W Jp l7 LN hH 4q zN YT 5g p2 vO LS rG QM 65 TV EP GD Qi hh nJ om aQ G3 E1 HR 8x wU KF id L0 2t 5c lB 81 IJ z0 Xn ub fk Bj 77 qe kW ht Hw XH YM Pu 2u b8 5p 22 P1 Ti Lb Aj bG NP Ed 2i od h7 3h 2G 7g yx wI mm nT NH fs cO ax ec dB 3c Qo oP bb A5 Sn gQ x2 R5 du 1Q dP 4f qh eI 1q lI ka Tz q5 iX hQ QH nM qH 3S Ve CE yh lM pk 8H Mm af XZ 4m eH 4F SW jl DX id QH Kd Rg 4G 6W mC yC Y1 ob Zf 5B uE hT 4O xy 5f k0 Ie Pr 3q Iw t1 PL JL qp qm 3q U3 vV 5T KK fW SM o9 AB Hv hz Oe Public Examinations Examined: Book Review – Questionmark

Posted by John Kleeman, Founder and Executive Director

Almost every country in the world uses public examinations in its education and admissions systems. This well-informed book examines the international use of such tests and exams.

They say that an unexamined life is not worth living. Perhaps a corollary is that if we do not examine well exams which impact positively and negatively so many young lives, our society will be worth living in less. This book is one of the last works by education quantitative researcher Thomas Kellaghan (he died while writing it); he and his co-author, Vincent Greaney have produced a beautifully written and insightful evidence-driven analysis of examinations round the world, that feels very much like a labor of love.

The book is available for purchase via the usual channels, but unusually and pleasantly is also available to e-read for free in the World Bank’s open knowledge repository.

Challenges the book deals with

I came to this book to try to help me answer a question I am speaking about at the forthcoming ATP Innovations in Testing conference: do tests and exams help make a better world? And in particular do exams help diversity and equity, do they give opportunity to the disadvantaged?

There was much in the book to help me. As in many areas, the book sits on the fence a bit – there are good arguments that exams can help equity and diversity, but also a recognition that they have weaknesses. And that even where exams are equitable, they cannot on their own change an unequal society.

Public examinations have a large impact on education systems and societies round the world. Within each country, there is often a big debate on the quality and utility of exams. But because exams differ so much internationally, there is little cross-border dialog. By giving a world overview of public examinations, this book has the potential to open perspectives to improve them everywhere.

What the book covers

The book crams in a lot in its 330 pages. There are chapters on the history of exams, how they are constructed and scored, whether and how they are valid, whether and how they are reliable, material on equity and malpractice and much more.

The authors are impressively informed. The index mentions over 100 countries from Albania to Zambia and there are examples of good practice and otherwise from a huge variety of systems. Everything is backed up by references, with several pages after each chapter of links to academic papers; in some ways this might be the most valuable aspect of the book.

There are also recommendations and analysis in various areas, for example how to improve the validity of examinations, and much discussion on the advantages and disadvantages on exams themselves and different aspects of them.

The book benefits from being published by the World Bank in its authoritativeness but perhaps because of this is very careful not to argue too forcefully in any direction.  Albeit it does set out many suggestions for good practice.  If the book has a weakness, it’s that it doesn’t look enough at exam technology, which in my view can transform public exam systems over the next decade or two.

How the book might help

I strongly recommend this book to anyone interested in public examinations. If you are involved in defining, reviewing or planning public examinations, then I think this is a must read. If you are interested or involved with assessment, it’s a very desirable read.

To summarize the book’s conclusion, the authors concur with a 1971 observation by Roach that “examinations had won their place, not because they were popular, but because they could not be dispensed with. The question … was … not whether they should be retained or abandoned, but how their efficiency could be improved”. The book suggests that those in charge of exams should move away from a “We’ve always done in this way” mindset and consider learning from international experience to put in place the most effective exams to aid society. If a few people listen and act on this, then the book could justify itself hugely by improving the lives and life chances of millions.

View more book reviews at questionmark.com/questionmark-book-review-series/

John is the Founder of Questionmark. He wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software development for over 30 years and has also participated in several standards initiatives.