Making decent progress at work

Dear Diary,

I haven’t been posting much about the stuff I’ve been up to at work.

I finished an application for the CANTAB research grant. My friend BB told me about it quite a while back and I set the deadline in my agenda just in case it’d be interesting. When the deadline was approaching, I looked into it again and had an idea for a project. The grant is not really for a lot of money (either $1,500 or 650€ depending on which prize you win) but you get to use the CANTAB test suite for a year. That’d be the interesting thing about it. I sat down to write up my ideas. Once I had a decent draft, I showed it to my supervisor and told him what the grant is. He had no reason not so support it so he helped me fine-tune a couple of things and wrote a great recommendation letter for me.

This week, I spent some time working on an abstract for Psychonomics. It’s a conference that’ll be held in Boston on November. The deadline for submissions is June 1. I want to submit something and finished the abstract today. The submission process is a bit tricky and I’ll have to figure that out tomorrow.

I have also been working on the pre-registration for an experiment. Two of the other co-authors have given their final “okay” and I hope that my supervisor will give me his blessing tomorrow. Then I can pre-register that study. My RA is in Paris until Monday – should could start collecting data for the project next week Wednesday. That’d be great.

I also finalized the code and all the paperwork for yet another experiment. She will also start collecting data for that when she’s back from Paris.

Tomorrow, I’ll spend some time analyzing the data for another project that she has been collecting data for in the last three weeks. There are a couple of complications because we had to switch from participants that signed up for course credit to those that signed up for money. It’s too late in the year now and virtually everyone has all their course credit so they are not signing up for studies anymore. But that means that I first have to check that there is no differences between the two “groups” of participants. Only then can I treat them as a single group. I did some basic checks today and it looks good. I have to check in a bit more detail and discuss it with my supervisor tomorrow. Best case scenario, I can use the data as it is now. Otherwise, my RA has to collect a bit more data next week. That wouldn’t be too bad either.

Anyways. All in all, things are going well. There are huge piles of work ahead of me. I’ll have to survey all the data a little bit and then make some strategic choices in the order in which I work on them. I’ll plan that once I have a better idea of the options that the data from the different experiments give me.

Stones are still rolling

Dear Diary,

all my projects are slowly creeping along. There are some issues with some of the CPR equipments but the warranty still covers it so the company will (hopefully) fix everything soon.

I’m waiting for someone else to send me some code that I need to finish up two other experiments. If I get that code later today or some time tomorrow, I might be able to finish everything up this weekend and have the experiments up and running early next week and make sure that data can be collected while I’m in Sweden. That’d be ideal. But if it doesn’t work out, it also wouldn’t be dramatic. We’ll see.

I’ve also been working on the pre-registration of the other project and sent another draft of the proposal out to the co-authors yesterday. One of the co-authors is my friend and roommate BB. But he’s in Berlin this weekend (until Tuesday actually) and I hope he finds some time in the train to look at the stuff and more or less gives his okay. I hope to be able to finalize the pre-registration early next week before I head out to Sweden. But I get the feeling that this will be a bit tricky to pull off in time. We’ll see.

So, I guess: We’ll see.

Getting the stone(s) rolling

Dear Diary,

this week, I was reasonably productive.

I finished up coding one experiment and the data collection started yesterday evening. My research assistant (RA) CS is doing the data collection. We won’t collect data next week because students are taking their resit exams.

My other RA is collecting data for another project. It’s a two-session experiment: the first session happened this week and the second will take place in four weeks. So things are on track but I don’t need to actively work on this yet.

Then I’ve been planning two more experiments. I have received confirmation from the ethics committee that I am allowed to run the experiments. That’s the first step. Then I met with a programmer yesterday who’ll help me write the code for the experiments. He’s very good and super nice and I should have the code early next week. Then I can test it during the week and then start data collection the week after that: on May 2.

That’ll be a short week because I’ll go to Germany on May 4 in the evening. The next morning, we’ll go to Sweden and I’ll be gone for the rest of the week and the week after that. Therefore, I want to make sure that everything’s up and running and I’ve tested everything. Because then my RAs can collect data while I am gone. If all goes well, I will be on vacation and when I get back, I’ll have data from three experiments!

And then I can dive into analyzing the data and writing stuff up! The plan sounds too good to be true so I’ll see how it actually works out. :D

On top of those projects, there is one more project that I’ve been working on this week. For that project, I’ll pre-register a study for the first time. The idea is that you write up your hypotheses before you start data collection and make them public. You also have to explain in detail how exactly you are going to analyze the data and all that stuff. I wrote a first draft of that “proposal” this week and sent it out to the people that are involved in the project. Based on their feedback, I hope I can finish up the pre-registration next week so that I can start data collection in the first week of May.

So, yeah. Lots of planning and organizing this week. But it’s going well. When I get back from Sweden, I’ll stop collecting data for a while and focus on analyzing and, most importantly, writing. I really need to make sure that I put all this stuff that I am doing at the moment on paper.

Work things

Dear Diary,

last week was pretty busy. I spend half a day with a master student that I am supervising. He finished data collection and I helped him with some programming basics to pre-process his data and do some basic steps. We basically checked to make sure that the manipulation worked and that we have the data in a format that is easy to work with for him.

He doesn’t really have much programming experience so I made sure to explain things to him and took my time doing things in a way that’s easy to understand (rather than quick to do). I could have done it in 90 minutes myself but instead we spent about four hours on it. But I think it was good: he learned a lot and now I know the data is a in a good format and he can focus on doing the analyses and then I can double-check his work.

Then, I also had two meetings regarding another project: I set up an experiment and organized the data collection for a project with my friend and colleague ES. I made some plots and pre-processed the data so that she could do the analyses more easily. She ran the analyses and the results are very, very interesting. We are the first to study this particular effect over a long time course. It’s a very boring task and we made people do it for an hour a day, four days in a row. We’re interest in how they adapt to the task and how the learn and how exactly they get better.

It’s interesting because the pattern of results changes quite a bit over time and no-one every has people do this task for more than an hour. So that’s pretty cool: we’re the first to show that the assumption that people won’t change their behavior is wrong. We met with ES’s supervisor to talk about how to write up the paper. I won’t be involved in that but will make the plots for the paper. So I don’t really have to do any work for it as of now. But I’ll be the second author on the paper. Which is pretty neat!

Then, for the paper I am writing with my supervisor, I am still waiting for him to send me the data so I can re-create his analyses and conduct two more. I sent him a draft of the Introduction and am waiting for his feedback. So there’s nothing I can do at this point.

On top of that, I am also planning two more experiments which I’ll focus on during the next week. One of them just needs to be programmed which is quite easy (I think…) and then my RA will collect the data. The other one, I want to pre-register. Which is a bit more work but also really cool because I’ve never done that before. Then the RA will collect the data.

For the other project, my other RA is starting another round of data collection tomorrow morning. Then we’ll have to wait for four weeks for people to get back and then we can analyze those data and think about how to write up the project.

So, yeah: there’s plenty to do. :) It’s quite fun but I also feel like I am reaching the limit of how much I can maintain in my brain at the same time…

A bit of a strange week

Dear Diary,

on Tuesday, I had a horribly unproductive day. I just couldn’t really focus. Eventually I gave up and just went home. Five minutes later, NB arrived and felt exactly the same. We decided to just draw the curtains and watch a movie. We googled a bit to find one we might want to see and checked out multiple lists of “Best movies of 2015”. There were a couple of movies that were on almost all lists and most of those we had already seen. Apart from Inside Out. It looks like a kid’s movie with an interesting setting:

The reason we ended up watching it was that it was, to date, the highest rated Pixar movie on imdb. Which is pretty amazing to me. Pixar made a bunch of great movies that are also appealing to adults (Up and Wall-E, just two name two) and this one had higher ratings than those?

And I have to admit: it was pretty great. It was a simple enough story but the way it was told was great. Many, many jokes and references in their that were definitely put in for the parents, not the kids. Overall, very enjoyable movie for the whole family.

The rest of the week until now (Thursday night) was a bit better: I was more focused and worked more but I was kind of stuck on the things that I needed to work on. I worked on analyzing two datasets. In one of the, the experimental manipulation just didn’t work and I have no idea why. We had 47 people study a set of words and tested them later and manipulated a set of the words in a way that should make those words harder to learn than the others. But there was no effect at all. I looked at various aspects of the data and can’t quite figure out why it didn’t work. One of the things I know is that it was just way too easy. But even if I look at only the data from the first couple of minutes, I see no effect. Strange.

The other issue is a data analysis for a conference paper that I want to write. I made some plots and started writing the paper. Based on the plots, I assumed the two groups that we assigned people to to differ in performance. But when I actually did the statistical analysis, it turned out they weren’t. There is quite a bit of variation between participants (not all participants learn equally well, of course) and also between items (not all items are equally difficult) and the analysis I did does not take that variation into account. So now I am working on a more complex analysis using linear mixed effects logistic regression to see to which extent I can account for that variance. And, more importantly, if there is a significant difference between the groups if I factor out the differences that I am not interested in (i.e., between participants and items).

I wasn’t able to do it properly yet. I think. It’s pretty complicated.

The problem is that I don’t know how to proceed writing the paper if there is no difference between those two groups. The way I need to tell the story kind of depends on that. So that’s a bit frustrating. Especially because the deadline for the paper is next Friday. Now I am a bit worried that there is no paper to write if there is no difference between those two groups even though we expected there to be one. My supervisor was sick today so I couldn’t really talk to him about it. I hope he’s back tomorrow or we can skype or something. I need to figure how what to write because there isn’t much time left…

Lots of new data!

Dear Diary,

while I was in Heidelberg, the data collection for two projects was completed. One was done by the research master student that I am supervising and the other by my RA.

I have things that are higher on my list of priorities but it takes me a lot of self-control to not go and look at the data. Exciting!

For the research master student, I’ll try to not do too much. I find it kind of hard because I can do in 2-3 hours what he’ll work on all week. He’s now starting to teach himself basic programming skills so for him it’ll be quite a bit of work. But it’ll also be a very good learning experience. So I’ll probably try to find time next week or the week after to sit down with him for four to five hours and walk him through the basics and help him get the data in a format that he can more easily work with. From there, I’ll let him do the work himself and then check up on what he did to make sure it’s correct.

The other dataset I’ll analyze myself. The RA doesn’t have any programming skills and is also not interested in learning it, I think. I’ll ask her again whether she wants to learn the basics or not in which case I’d invite her to the session with the master student. It’s not something that would be part of the job we’re paying her for but she might be interested in learning it to expand her skill set.

The analyses should be relatively straightforward. I’ll probably need an hour or two of preprocessing and maybe two or three hours to analyze the data and create some plots. Then the results should be relatively clear and we can talk about how to write them up. But I won’t have time to think about that before the second week of April. Should be a fun project, though, and I can’t wait to see the data!

Conference in Heidelberg

Dear Diary,

on Sunday, we drove to Leer and parked the car there. (The train station in Leer is sketchy as fuck…). Then we took the train to Heidelberg. Which is quite far away. From the station in Heidelberg, we walked for 20 minutes to our hotel in the Altstadt. We checked in at around 11 pm.

It was my supervisor and our research assistant and student SM. The conference started the next morning and the first session was at 8:40. I was the last speaker in that session and also had to chair it. Which meant I had to be there at 8:30. Which was quite painful for me.

My talk went well. And the other talks in the session were also really good. It was a shame that almost no-one saw our talks. It was kind of stupid of the organizers to start the conference so early. Most people don’t live far enough away to already arrive the night before and traveled on Monday morning. So a lot of people skipped the first (two) session(s) and arrive later. At the very first talk in my session, the audience was basically empty: it was the other speakers and two more people. (The guy took it well and said, “let me take a picture of the audience to show to my mum how relevant my work is.” :D ). By the time I was up there were about 15 or maybe 20 people.

The advantage was that I was “done”. I didn’t need to worry about my presentation anymore and could just sit in on all the talks and browse the program. Which was surprisingly extensive. There were four sessions with talks each day. Each session had four talks and at each moment, there were 12 parallel sessions. That about 190 possible talks a day (20 minutes each). There was a keynote speaker at 12 each day and a one-hour lunch break followed by a one-hour poster session. Each day started at 8:40 am and the last talk ended at 6:20 pm. That is to say: it was pretty intense.

I didn’t really come across anything that was directly and extremely relevant to my own work. But I did come across a lot of interesting ideas and got to talk to a bunch of people that provided helpful comments and that I could provide some inspiration to as well. It was pretty great.

One of the things that I took away from the conference is that I might want to look into the literature of “educational games”. It’s a whole field that I have not really thought about at all. But they do a lot of research on how to make the games more enjoyable for people to play and maximize what people learn from those games (and also how to track and quantify learning). It’s kind of weird that I never really thought about this before but that’s a good way to think about our learning software and to possibly extend it in the future.

The other thing is that I started planning another project that should be very easy and straightforward but will be interesting. In the current,  big project, we estimate how well people can learn facts and get their grades. There is a small relationship between those two measures which is great. Especially because grades are not correlated at all with IQ (in the same sample). We also measured motivation. That is, a specific questionnaire that assessed “study-related motivation”. Some of those sub-scales are also related to grades (as expected). But the motivation scores are not at all related to how well someone learns facts.

Which makes a lot of sense: there’s no reason to assume that someone is better at some random task in the laboratory just because they are motivated to get high grades and have the self-discipline to get them.

I do think there’s an influence of motivation on fact-learning, though. Some people are just more motivated than others. And some people are just better than others. But it’s difficult to tease the two apart: if someone gets high scores, we don’t know to which extent that is because they are very good or because they are highly motivated.

Therefore, I’ve been thinking about ways to experimentally manipulate motivation in the learning task. There are several ways to do it. The most obvious is to pay participants based on their performance: you have 20 minutes to study these words. You get 50 cents for each correct answer on the test. We could compare that group with a group that gets a base rate: everyone gets paid the same. (And maybe we can even add a third group that does it for “study credit” – no payment at all.)

I think that’s the simplest and most straightforward way. Alternatively, you can also use “social comparison” to motivate people. That is, you can either tell them “you’re doing better/worse than the 10 people before you”, for example. Although that is a bit more tricky because the effect of such a manipulation depends  a lot on the exact situation and also the person. (Some people might get worse if you tell them they’re doing very well because they have no reason to try harder. Others might either give up if they are told they are doing poorly or they might work extra hard to try and keep up with “the others”. So it’s less predictable.)

I’ll be very busy with a bunch of other things in the next two weeks. But I’ll talk to my research assistant and ask her to look into the literature. There are a bunch of studies that manipulate motivation during learning. I’ll ask her to compile a small overview of the techniques usually used. I think paying people for their performance is the best way to do it but maybe a literature search will reveal a better idea of reasons why my idea is a bad idea.

It might also be tricky to get this through the Ethics Committee because variable pay-offs are not as easy. I think it should be possible (and otherwise, I know how to do it without the Ethics Committee at Psychology but I’d prefer to do it the official way….)

Anyways – very productive and stimulating conference. Heidelberg is also very pretty. :)

Experiments are running better than expected

Dear Diary,

both experiments are now running. The data collection is going a lot faster than I thought it would. Which is awesome. One experiment takes 60 minutes and is run by a student assistant in a room in which we can test five people in parallel. She opened a bunch of time slots throughout the week for people to sign up and they fill up fast. We were aiming for about 40 participants and she said that it looks as if she can collect about 40 this week alone. The room is booked by someone else for parts of next week but I asked her to open more time slots then: if it’s going that well and people are signing up like crazy, we might as well collect a couple more: if I end up getting data from 60+ participants in 5 or 6 days, even better. She also doesn’t mind because she’s paid for it and I programmed the experiment in a way that it basically runs itself (I always try to minimize or eliminate any interaction between the participant and the experimenter if possible) so she can sit in the room and study.

The other experiment is run by a master student I am supervising. He’s using a room with 16 computers – so he can, in theory, test 16 people in parallel. He’ll test about 60 this week but it’s a two-session experiment so we’ll see how many of them come back and do the second session. We can only really use their data if they come for both sessions so we need to plan for some overhead. For this experiment, we’re paying the participants so we don’t want to just collect a whole bunch more just because we can: that might get expensive.

Either way – this is great. There’ll be a lot of data waiting for me when I get back from the conference next week!

Two more experiments ready to run

Dear Diary,

I spent quite some time last week getting two more experiments up and running. Data collection is going to start tomorrow for one and on Tuesday for the other. Exciting stuff. The data collection will be done by students – I will just supervise the first sessions and then they’ll do the rest.

I am excited to get in more data. But I think I’ll have to sit together with my supervisor and talk about the data I already have and how to use it for papers. I feel like I should stop collecting data towards the end of April. Because I already have a bunch of data that I haven’t written about. And for my thesis, only papers count – not the number of data sets I still have on my computer. So I think my focus should shift a bit in the next couple of weeks and I should go into full-on writing mode.

I am actually looking forward to that. The weather is going to become nicer – I’ll spend part of the day reading papers, and then writing up my own work. Working with the data and putting my work out there. I feel like I spent the last 18 months or so collecting a lot of data. And I looked at the data myself but haven’t had the time to write stuff up and share it with the world.

If the projects I have running now all work out more or less and I can write up most of the stuff I have in mind, I think my thesis is going to be pretty good!

Work has me again

Dear Diary,

I am fully recovered and am back to work. And there’s lots and lots to do!

I have taken a look at the data that my RA collected. She collected data from 20 participants so we can take a look at the effects. This required me to write quite some code to manage and merge the data and create a whole range of visualizations. It’s a complicated project with a very complex data set.

What we’ve learned from the data of the 20 people is: (a) one week is not enough for people to forget how to do CPR in any meaningful way (not a big surprise) and (b) we need more data.

Therefore, she’ll collect data from 20 more participants. But that won’t happen until the beginning of April. So the project is “on hold” until then. Which is not a problem because I’ll have plenty of other things to do. My initial hope was that there’s enough in this data to say a bit more so that I can write a short conference paper about it. The deadline for that paper is on April 8. So I won’t have the data for this project in time. But that’s not a problem because I have data from another project that I can write about. I put a rough outline of my ideas on paper today and will refine it in the next couple of days.

I am setting up another project with another RA that’ll collect data for me for a study in which we want to look at interference effects in fact learning. That is, we want to take a closer look at the effect that the words in set of facts that you study have on each other. I had to organize a few things but everything should work out: data collection will start next week.

Then, there’s a research master student that looks into the effects that the time of day (and your “chronotype”) has on studying factual knowledge. I am co-supervising the student. I spend about 5 hours in total this week on his project and we will finish everything tomorrow. He’ll start data collection on Tuesday.

Thus, until the end of the month, there are two new projects for which I’ll have data to work on. The timing is quite nice: I made sure that both those projects are on track and that data collection for both starts next week. Because I’ll still be in Groningen then and can supervise the process and fix things if necessary. The week after that, I’ll be in Heidelberg, presenting my work at a conference. Ideally – if all goes according to schedule – a lot of new data will wait for me after Easter. The reason I wanted to get this going now is because otherwise I’d have needed to wait until after Easter to start the data collection.

Now the RA and the student will collect data and I’ll be working on preparing my presentation and writing the conference paper. Then, once the presentation is given and the paper is submitted (i.e., in early April), I can focus on analyzing the new data.

But that’s why this week is a bit hectic: I need to get this stuff done so that data collection can start next week. But it seems to work out. :) It feels good to be productive again.