Evaluating replicability of laboratory experiments in economics

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

Evaluating replicability of laboratory experiments in economics.pdf




來源:科學月刊,2014 年 5 月號。

諾貝爾經濟獎於 1969 年設立,在 48 年的頒獎中,總共有 24 年均有當時在芝加哥大學任教或研究的經濟學家。沙勒兒是第 34 位獲此殊榮的芝加哥大學學者,但他的理論卻完全與「芝加哥經濟學派」(Chicago school of economics)的理性經濟學說相反!


    ESA-announce WZB Berlin oTree Hackathon 26-29 October

    The first WZB oTree Hackathon, to be held 26-29 October 2017 at Berlin Social Science Center (WZB) in Berlin.

    Jointly sponsored by WZB and LEEPS lab, University of California Santa Cruz, the hackathon is intended to bring the sprawling oTree community together. Programmers and researchers will connect and exchange ideas, and will extend oTree’s capabilities. Participants will form teams with 2-5 members and intensively develop a project helpful for their own research and for the oTree community. The event is targeted both at seasoned programmers who want to develop new oTree techniques and economists interested in using oTree in their research. Also, we will be organizing a crash course for folks unfamiliar with oTree.

    If you want to find out more about the WZB oTree hackathon (or what a hackathon is) visit our Hackathon website at https://www.wzb.eu/en/events/wzb-otree-hackathon

    Registration is open till October 15. If you want to atted, please send an email to gebhard.glock by October 15, indicating your name and institution.

    Accomodation is available at Motel One (An der Urania 12, 10787 Berlin) and can be booked until September 28 by sending the hotel reservation form to the hotel. Accommodation costs during the hackathon can be reimbursed for graduate students, and food for all participants will be provided at WZB. Airfare or any other transportation costs cannot be reimbursed.

    We hope to see you soon in Berlin!

    Perceived social presence reduces fact-checking

    Jun, Y., Meng, R., & Johar, G. V. (2017). Perceived social presence reduces fact-checking. Proceedings of the National Academy of Sciences, 114(23), 5976–5981.



    參與者被分成兩組,其中一組人在登入新聞網站後,會見到自己的名字顯示在網頁角落,在此將這組稱之為「單獨」組。另一組人在登入後,除了自己的名字外,還會看到另外 102 個正在線上的其他使用者的名字,稱之為「群體」組。

    接下來的實驗更神奇了。研究者把原本的「新聞網站」改成「Facebook」,也就是讓參與者在Facebook的介面上讀這些新聞標題並判斷真偽。結果當介面換成 Facebook 這種社交網站之後,「單獨」組選擇「事實查核」的比例竟然降得跟「群體」組一樣低。

    最後一個實驗中,研究者想知道人們的警覺是不是造成「群體」組事實查核比例較低這個現象的原因之一,於是便要求參與者在開始讀新聞之前先回想自己過去和現在的職責、義務與責任(duties, obligations, and responsibilities)。這麼一做果然讓「群體」組選擇「事實查核」的比例增加到跟「單獨」組一樣高!


    On the performance of the lottery procedure for controlling risk preferences

    Berg, Joyce E., Thomas A. Rietz, and John W. Dickhaut. “On the performance of the lottery procedure for controlling risk preferences." Handbook of Experimental Economics Results 1 (2008): 1087-1097.
    在 2008出版的 “Handbook of Experimental Economics Results " 指出, (在實驗中使用 lottery procedure的時候) 風險愈高的實驗, 風險偏好的影響愈大….
    ==original abstract==
    This chapter describes the lottery procedure for inducing preferences over units of experimental exchange and show how it is supported by several experiments on behavior in simple contexts. This chapter reviews various evidence from a set of paired choice and pricing tasks designed to determine whether subjects’ revealed preferences over gambles are consistent with attempted risk preference induction. There is strong support for the performance of inducing when subjects choose between paired gambles. Subjects induced to be risk seeking nearly always choose the riskier gamble, while those induced to be risk averse choose the less risky one. There is similar support for pricing gambles, (有哪些 pricing gambles??? 待查) but the strength of the effect is a function of the variance of the gambles. This is consistent with other experimental evidence about the importance of saliency. Risk preferences matter little when there is little risk! As risk increases, risk preferences should become more important in the experiments. Subjects appear to price gambles more consistently with their induced risk preferences as variance increases.

    Inducing risk neutral preferences with binary lotteries: A reconsideration

    Harrison, G. W., Martínez-Correa, J., & Swarthout, J. T. (2013). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior & Organization, 94, 145-159. [working paper 版]
    lottery procedure 是實驗中避免 risk-averse 的受試者, 影響經濟行為的一種控制方法, 但是否有效, 好像還沒有一致的結論…
    On the performance of the lottery procedure for controlling risk preferences  一文中指出, 只要 risk 不要太大, lottery procedure 是可以 induce 出需要的偏好
    ==original ABSTRACT==
    We evaluate the binary lottery procedure for inducing risk neutral behavior. We strip the experimental implementation down to bare bones, taking care to avoid any potentially confounding assumption about behavior having to be made. In particular, our evaluation does not rely on the assumed validity of any strategic equilibrium behavior, or even the customary independence axiom. We show that subjects sampled from our population are generally risk averse when lotteries are defined over monetary outcomes, and that the binary lottery procedure does indeed induce a statistically significant shifttowards risk neutrality. This striking result generaliz es to the case in which subjects make several  lottery choices and one is selected for payment.

    Team Incentives and Performance: Evidence from a Retail Chain

    Friebel, G., Heinz, M., Krüger, M., & Zubanov, N. (2015), “Team Incentives and Performance: Evidence from a Retail Chain." American Economic Review, Vol. 107, Issue 8 — August 2017. * [working paper]

    Guido Friebel, Matthias Heinz, Miriam Krueger and Nikolay Zubanov
    In a field experiment with a retail chain (1,300 employees, 193 shops), randomly selected sales teams received a bonus. The bonus increases both sales and number of customers dealt with by 3 percent. Each dollar spent on the bonus generates $3.80 in sales, and $2.10 in profit. Wages increase by 2.2 percent while inequality rises only moderately. The analysis suggests effort complementarities to be important, and the effectiveness of peer pressure in overcoming free-riding to be limited. After rolling out the bonus treatment, and control shops’ performance converge, suggesting long-term stability of the treatment effect.
    Full-Text Access | Supplementary Materials
    eTOC for American Economic Review Vol. 107, Issue 8 — August 2017

    GIMS, an open source market software

    S. Palan, GIMS-Software for asset market experiments. J. Behav. Exp. Finance 5, 1–14, (2015). Medline doi:10.1016/j.jbef.2015.02.001 (可免費閱讀)

    ==by YNY==

    GIMS 是架在 z-Tree 上,專門用來跑財務資產市場 (又稱 double auction asset market) 的實驗平台軟體,採開放源碼 (open source) 授權。

    此文亦介紹、比較了其它相關的財務市場實驗平台軟體,參見文中的 Table 1。

    • EconPort MarketLink(based on Java and experiments can be run over the internet)
    • Flex-E-Markets(not open source)
    • GIMS(based on z-Tree)
    • jMarkets (open-source software based on Java)
    • Rotman Interactive Trader (非免費)
    • SoPHIE Labs (非免費)

    Continuous double auction market interface

    Evaluating replicability of laboratory experiments in economics

    C. F. Camerer, A. Dreber, E. Forsell, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, J. Almenberg, A. Altmejd, T. Chan, E. Heikensten, F. Holzmeister, T. Imai, S. Isaksson, G. Nave, T. Pfeiffer, M. Razen, H. Wu. Evaluating replicability of laboratory experiments in economics. Science, 2016; DOI: 10.1126/science.aaf0918


    終於有人系統性地檢視經濟實驗, 雖然只挑 AER 和 QJE 所發表的, 但經得起「重覆實驗」來驗證結果的研究, 才符合科學的精神…