Recently it was discovered that somebody in my research area, individual differences in social behavior in animals, likely committed egregious errors, if not outright fraud, throughout his career. While I dont consider myself a behavioral ecologist, my research often dovetails with this group. I was therefore shocked that specific individual researchers were using the case to lambast the study of individual differences in animal behavior as a whole, comparing the act of a single researcher to the replication crisis that plagued the whole field of social psychology recently.
At the core of the controversy was a common dynamic in nearly all collaborations. Some collaborators (usually with a computational skill not shared by others) receives the data, collected by another person, and analyzes the data with no extended experiences on how the data was collected. For myself, getting an excel file and a short description of how the data was collected, is never sufficient to truly make use of the data. A lot of my work is observational, looking at patterns of social interactions in semi-naturalistic settings, and seeing how those patterns change over development, or in response to group changes. Such studies require the skill of keen observation, a skill that can only be cultivated through months of rigorous training and reliability checks with others, and with yourself.
I truly believe that the best science will be produced when all researchers involved in a project have intimate hands-on expertise with the specific organisms they are researching, and that the dangers of collaborations lie when such expertise isn’t shared, in particular, because it leaves innocent individuals vulnerable to being taken advantage of by others. My experiences doing research has shown that checking for statistical irregularities in data is different from checking for biological or behavioral inconsistencies, and that just because data meets the bar for statistical analysis doesn’t mean they meet the bar for behavioral significance. Furthermore, it is those who have extensive experience with their organisms who are the best suited for assessing behavioral significance. But above and beyond all, I am a firm believer that all studies should begin and end with the organism. It is the organism that is the ultimate arbiter of truth in our science, and neither our trust in our colleagues, algorithms, assumptions, or hypotheses, should bypass our tendency to confirm our suspicions by observing the organisms ourselves.
All data is a limited and indirect summary of the system you are observing. No data is ever “raw”. Each data set is collected under a particular set of circumstances, and is always partial to the fallacies of human judgement and design. Even tools that attempt to bypass such human fallibilities, such deep learning, are designed to be deeply influenced by the structure of information in the world that in turn is a reflection of human effort and judgement. For better or worse, human judgment is going to be the producer and consumer of data, as no outside force can divine us with pure truth in spreadsheet form. So what can we do when such human judgment fail us? How can we prevent such failures from reoccurring?
One solution proposed is to keep sharing data, to make data as open and transparent as possible. While I am in support of such efforts, I feel they are barely sufficient to protect the integrity of data from those with strong careerist motivations. Already stretched thin researchers dont have the time to be policing others work constantly, and I worry that the vagueness of “open science” will lead to a more segregated class of “data collectors” and “data analyzers” and lay the responsibility for data integrity with those with the time and motivation to check data who themselves, and who are themselves also fallible to bias and nefarious motives.
I am not under any illusions that I have a solution to this issue, but one thing has consistently troubled me is the willingness of researchers (including myself) to use data when they have little understanding for how the data was produced. I used the word produced here intentionally, human labor produces and creates data we expect to approximate real observations on real systems. Thus, the conclusions that can be drawn from data are ultimately a product of the processes the way the data is produced, and understanding the production process is key to understanding the limitations of a dataset. In many cases potential confounds, misinterpretations, and limitations of the data can be clearly seen by those intimately familiar with the system the data is supposed to be describing, but completely lost to others unfamiliar with the system. The idea that researchers need to have some grounded experience in the way data is produced, and the system they are investigating, often directly conflicts with the proposed solutions offered by the strong advocates of open data, who believe that transparency alone may be enough to prevent bad data. However, it is of my opinion that researchers who have intimate familiarity with the systems they are investigating are collectively the best suited to telling when data suggests something is so out of character to suggest potential errors, or exciting discoveries.
Which brings me back to the situation in animal personality and individual differences in social behavior. No, this field itself is not in crisis. But yes, we in animal behavior, comparative psychology and behavioural neuroscience need to be more reflective about how we approach behavioral data. Frankly put, those analyzing the data from specific animal systems should at least have extensive experience with those animals, and first hand at least and at best have a role in collecting the data themselves (if possible run the experiment in your own lab before jumping to conclusions with shared data).