A collection of post-mortems
Table of Contents
Published on: August 6, 2024
This page is still under the development.
We’ve just completed the 2024 Summer Institutes in Computational Social Science. The purpose of the Summer Institutes is to bring together graduate students, postdoctoral researchers, and beginning faculty interested in computational social science. The Summer Institutes are for both social scientists (broadly conceived) and data scientists (broadly conceived). This summer we had a mixture of in-person institutes and virtual institutes across the world. In addition to SICSS-Duke, which was organized by Chris Bail, there were … partner locations run by SICSS alumni.
These post-mortems describe, for each site, a) how the Institute was run, b) what each site think worked well, and c) what each site will do differently next time. We hope that this will be useful to others organizing similar Summer Institutes or future organizers of SICSS sites. If you are interested in hosting a partner location of SICSS 2025 at your university, company, NGO, or governmental organization, please read our information for potential partner locations.
This page includes post-mortem reports from all locations in order to facilitate comparisons, as well as an overview of key themes and takeaways. As you will see, different sites did things differently, and we think that this kind of customization was an important part of how we were successful.
Skip to:
We’ve divided the post-mortem into 5 parts: 1) outreach and application process, 2) pre-arrival and onboarding, 3) the first week, 4) the second week (group projects) and 5) post-departure.
The second edition of SICSS in Poland, in the form of the SICSS-AMU/Law, drew in large part on the experience of the previous, 2022, edition. Three co-organizers of the previous edition were again involved in the preparation of the school, and we recruited seven new people (3 responsible for outreach and marketing, and 4 Teaching Assistants). Already at the recruitment stage, we were guided by the lessons learned from the first edition and composed a team to improve those areas, where we saw room for improvement.
The first change was in the advertising of the school. Based on previous data and materials, it was necessary to create more modern, unique, attention–grabbing and encouraging content on our social media. It turned out that it is more efficient to work in a small group (3 persons) rather than give this field to a group of students who are less related to the core team of the institute. In this edition we were supported by two students who we knew personally from different university activities. They supported our operations’ director in all marketing activities before and during the institute. However, even in advertising, there is always the kind of work that you cannot completely outsource to third persons. There were some information that only core team knew about and we learned along the way that we need to keep these two marketing people in the loop.
As in previous edition, our plan was to gather young scientists from Poland and surrounding countries through an extensive advertising campaign, for which we used not only social media, but also e-mail. Following the experience of the previous edition, we decided to focus on our Facebook and LinkedIn fanpages, where we more regularly posted information about speakers, schedules, interesting articles and the team. Another change we introduced was content planning – we managed to make a list of possible ideas for social media posts and schedule the posts so we did not have to brain-storm ideas every single time there needs to be something posted online. One thing we learned is that the more ambitious we want our social media to be, the more creative work there is to be done. We wanted to keep everything integral as far as esthetics (templates, logos) are concerned. This generated much more work, but it seems it paid off well. Since we have no faculty marketing team, we needed to do everything on our own.
The traditional way of promotion – through the University website, Facebook, LinkedIn and e-mailing appeared to be relatively effective, but not enough for us to gather a satisfactory number of participants in time. The information about the event was also disseminated on topical groups of scientific events, conferences and seminars. Therefore, after inner consultations, we decided to go with paid advertising, targeted according to the criteria that we declared to consider in the application process (e.g. geographical location, legal/policy-making background). From the very beginning, learned from the past experience, we also conducted an intensive mailing campaign, as it is the only direct channel of communication with potential candidates. For that purpose, we used a database of e-mail addresses from selected countries based on publicly available sites, which was created as a part of the previous edition. In the effect, for three weeks we have sent around 10.000 e-mails with invitations to the Institute.
As a result, we received around 50 applications ranging from the US to Europe. That was enough to run an institute and make some selections. In order to select participants we used the following criteria: 1) research and teaching in the area of computational social science 2) contributions to public goods, such as creating open-source software, curating public datasets, and creating educational opportunities for others 3) likelihood to benefit from participation 4) likelihood to contribute to the educational experience of other participants, and 5) potential to spread computational social science to new intellectual communities and areas of research. Further, when making our evaluations, we accounted for an applicant’s career stage and previous educational opportunities. A factor that was eventually hard to assess is digital skills.
Finally, we selected 24 participants representing universities from: the United States, Estonia, Germany, France, Slovenia, Albania, Turkey and Poland, as well as public administration offices. At the beginning, two of them did not confirm participation, and after the Institute started, 2 resigned. It would be good to make sure that those who resigned in the first day had a chance to talk to us before leaving the institute (e.g. writing to those who did not show up the day after).Consequently, 20 people managed to finish this year’s SICSS-AMU/Law. This confirmed that it is always good to accept slightly more people in the first place.
Alongside the promotion and participants’ recruitment process, we were gathering a group of experts to join the school. While the legal background of our SICSS and the previous experience from 2022 (photos, stories, expert) were a strong foundation of our invitations we faced a number of challenges.
The first one relates to the time of sending invitations – we realized that while waiting for the final decision on our additional funding for the experts’ remuneration we were missing opportunities of “booking” experts’ calendar. It seems better to contact expert anyways, even when funding is not yet secured. Some were surprised to be offered remuneration and were prepared to take part in the event pro bono as well. We took care of the remuneration of the experts in order to, on the one hand, ensure an appropriate level of their involvement (e.g., preparation of a dedicated presentation, pre-event online meeting to agree upon the content etc.) and, on the other hand, contribute to the professional image of the SICSS brand, as well as of our Faculty.
The second one relates to not responding to our e-mails or messages on social media. While people we knew personally responded quickly, those who we approached for the first time used to leave our messages unanswered (~50%). Our guess is that approaching “big names” in the field without any direct contact ever before may not be the best strategy as they typically receive a lot of invitations and may simply be overloaded with work. While gender parity in the experts’ line-up was easy to achieve, we had difficulties with geographical parity. Despite our strong links with European researchers, researchers affiliated to American universities were much more responsive and enthusiastic.
The third one relates to not continuing conversation after initially responding to us or responding after a couple of weeks. This can be related to the reason given above. We learned, however, that this varies and no general conclusions should be made. We think that it is hard to grab one’s attention these days, and the best way is simply to have plan B with a number of different experts that are worth inviting.
Despite the above challenges, we managed to collect a diverse panel of experts that brough an exceptional contribution to the school.
To the selected group that confirmed their participation, various resources were indicated. This year we decided to send a brochure with all important information gathered in one file (e.g. how much time they should book in their private schedule to participate in the school, how to join our Slack, how to get engaged with Python, and how to get ready for the group projects). As far as the substantive preparation was concerned, we suggested online courses of Python so the participants with no-coding experience do not get confused on the first day. We also suggested to watch the video of Chris Bail on “Why SICSS?”, which seemed a like a good mental preparation for the school, although not many people actually watched it. This year we did not do a dedicated boot-camp, as it was not popular last time, and besides that we switched to a less-code approach towards programming (using pre-made code on Google Colab, using LLMs as a help in coding). As the next days showed, the participants could have better familiarized themselves with how Google Colab works and how to operate Python scripts. It seems, that the practical aspect of the school was not obvious for some participants that were (positively) surprised with the number of exercise and programming tasks. This, however, does not mean they were well-prepared to deal with them with ease – which is a similar conclusion to that of 2022. Perhaps, a clearer message on that issue could have helped in better understanding of the Institute and skills needed. Perhaps a good solution is to design an LLM-based game (see example) that introduces the basic concepts and also sets the right mindset needed to learn programming (e.g. preparing for multiple errors). We must admit, however, that there was a number of participants without any coding skills who managed to learn the ropes and produce exceptionally well-coded project in week 2. This proves that it is more about your determination and general digital skill, rather than knowing a specific programming language.
We encountered no abundant correspondence regarding organizational matters. As it was pointed out, we received limited response from the candidates accepted to the Institute. Among those who confirmed, there were a few that either did not show up at all or quit after day 0 (virtual get-together) or day 1. We received no contact from them. In such cases it might be good to get in touch and learn about their motives (which can be either personal or due to the Institute’s agenda). We felt, however, that those people took seats from those who wanted to participate but were refused admission due to the maximum number of participants reached.
We began the first week with a virtual get-together on Sunday evening. We abandoned a traditional introductory round, during which every person says a few words about themselves. Instead, we launched a Kahoot quiz and asked for their motivation, background etc. In addition, we made two short (ca. 10-minute long) speed dating sessions in breakout rooms (3 persons in each room) with random allocation of participants. It seems that the get-together could be longer, as an extra dating session would allow to have all students actually meet each other in a breakout room.
We followed the traditional framework of the two-week SICSS, meaning that the first week was a mix of lectures and workshops. The day started at 13:00 and lasted until 18:30-19:00. This was supposed to allow the participants to catch up with their professional and personal responsibilities in the morning. In addition, this was supposed to mitigate the Zoom fatigue. We did not link the substance of a workshop with what was presented by an expert. Last time, it proved to be exceptionally difficult to make sure the entire day relates to the same topic (e.g., due to last-minute adjustments in the schedule). In our case such difference in what speakers presented and what workshop was about did not cause any confusion.
Typically, the guest lecture lasted 1,5 hour whilst workshops including group exercises took 2,5 hours. In addition, we allocated approximately one hour for brown-bag lunches in the Zoom breakout rooms, where participants were invited to discuss their ideas for the 2nd week’s projects. The latter proved successful as we had three groups established already in the first week and two participants who decided to carry out the projects as lone wolves. Brown-bag lunches arguably contributed to the positive group dynamic as most participants stayed online for the whole time and took the opportunity to get to know each other better. However, some students reported that it is hard to combine during a brown-bug lunch 1) meal-preparation, 2) networking with other participants, 3) pitching and discussing of the group project ideas. In the middle of the week, after an interim survey, we decided to divide break into two parts: one devoted to actually having a break (personal matters), and one to networking.
At the end of each day, we had a 30-minutes long wrap-up which was sufficient to summarize the exercises and discuss problems that occurred during workshops. We noticed students appreciate when we ask extra questions about their struggles and personal difficulties. We also confirmed our intuition that Wednesday is typically a low-point day. Students realize how little they know and how much there is to learn, even basic stuff such as operating your Slack or Google Colab properly. At that point it is important to give a word of encouragement and show that it is everyone’s path and what they do is already an achievement.
The main focus of our Institute was investigating how social polarization influences law, however, from the very beginning, we aimed to recruit participants with diverse background. This also meant that the topics, guest speakers, and exercises had to be law-related and at the same time accessible to non-lawyers. Therefore, we covered the following topics: 1) natural language processing, 2) using large language models both via browser and API, 3) judgement prediction using machine learning techniques, and 4) statistical data analysis in dogmatic analysis of legal acts. All participants decided to apply large language models and natural language processing techniques in the second week, which clearly shows that these topics are truly useful for participants and devoting all workshops to this topic was a good choice.
All of our teaching assistants prepared exceptional materials for the workshops (Google Colab files). There is a number of improvements that boosted the workshops this year:
All lectures were generally programming-free and focused on the general issues related to application of computational methods or influence of social polarization on law. All workshops were about explanation and application of programming skills and turned out to be a great success for at least two reasons. The participants had opportunity to work hard (in terms of the difficulty of programming) by actively engaging in workshops and little relax from technical aspects during lectures by engaging only passively, every day. Workshops were divided into two part. During the first part, the participants listened how to implement computational methods using Python, during the second part participants were split into smaller groups to work together on activities (which are all posted on our website and Slack). The exercises were programming-oriented and consisted of smaller tasks. As most participants struggled with the basics of Python programming, we have been prepared for that and during all workshops at least three teaching assistants were available to help struggling participants.
This year a special focus was given to the issue of polarization. We decided that this can be approached both through expert lectures and workshops. One of the cases was that of the Polish criminal policy. During the expert lecture of Dr. Marta Nawrocka-Świętkowiak we analyzed how punitiveness of certain crimes depends on the political views of the governing party. In the following workshop we allowed students to analyze law provisions on their own, in particular in order to see whether criminal policy changes and how the changes reflect the approach towards certain crime types. The other approach towards polarization was taken on the European level. In the workshop on the judgments predictions of the ECHR we took an attempt to see whether there is any change in the rulings depending on the factors such as sex of the applicant. This also made a basis for one of the group projects in week two that confirmed that there is no gender-based polarization taking place in this regard. Finally, we invited a representative of an NGO “Computational Democracy Project” – Liz Barry – to show us how public opinion tools, such as Polis, can help learn what the society wants to say in the most important social and political issues. Using Polis by social movements can give light to statements the way they are pronounced by the respondents, thus – learning what people think in their own language. We had a chance to learn how this tool helped fight social polarization in countries where great divisions exist, and how it may help to fight difficult, politicly-driven debates over regulating vital phenomena by helping create consensus among people (such as the case of Uber in Taiwan).
During the SICSS-AMU/Law, three groups were formed and two lone wolves, each one realized one project. The groups prepared great and very advanced project with ready code and analyses. The first group investigated gender bias in ECHR judgements. The second group examined if it is possible to predict if given statement would be considered as offensive by court. The third group tried to design a tool finding connections between testimonies based on a common object (e.g. a person that was mentioned by different people).
Lone wolves prepared just conspectus of their studies which lacked a methodological foundation, but consisted of more general approach on what kind of technology might be helpful.
From the substantive point of view, the topics of the projects proposed by groups were very interesting and well-thought-out. Basically, the research could be continued and probably could end up with successful publications, because the conclusions already presented during the SICSS were really promising, and scientifically useful.
From the programming point of view, all groups faced similar problems. Both, during the projects and during the workshops (in the first week). Participants were generally very inattentive and made a lot of typos. As a result, correctly written code did not work, which is very frustrating and, unfortunately, can be daunting. Another problem was misunderstanding the data structure and trying to write code without a plan. The programmer should have got a clear idea to write a program. Otherwise, the effects will be dissatisfactory. However, most of these problems were solved by wide use of help of ChatGPT and the help of TAs. Maybe in the future, we should teach more about prompt engineering (for example by preparing bootcamp devoted only to ChatGPT assistance) or encourage participants to use such programming tools were AI assistance is present by design. In this year all participants were using Google Colab, where Gemini is implemented, however this feature was released just before our SICSS edition, and we did not have enough time to train participants.
Besides seeking help from ChatGPT, participants actively reached out to TAs to discuss issues and find solutions together. Materials prepared by TAs also proved to be very helpful (e.g. step-by-step instructions, plain language). Compared to our previous edition, these materials were significantly more useful for replication, and the TAs themselves were more open and accessible. Duty hours were relatively popular, but most issues were resolved either through ChatGPT or by contacting TAs outside of duty hours.
The collaboration within the groups was particularly good. Members used two channels of communication. Most of the discussions took place via Slack chat. Sometimes, when needed, the participants organized video meetings at Zoom.
We highlighted four top participants based on their commitment, participation in discussions, scientific excellence, and progress made throughout the entire school. Additionally, we decided to award the best project among those presented on the final day. The prize will include support in programming and computational methods provided by our data scientist.
Using Slack as the main communication channel was a very good idea, even the free version was sufficient. The second channel of communication, more direct, was Zoom. To simplify usage and avoid problems, we organized every meeting and duty hour under the same link. Maybe one software for both chat and video meetings is a nice idea for improvement.
The Institute was concluded in a friendly atmosphere with a noticeable vibe of success among those who made the biggest progress coding-wise. We suggested joining the SICSS world community to reach out to the broader group of scientists who apply computational methods in their work. We provided participants with personal certificates of completion of the Institute.
After the school ended, participants have been staying in touch with us and among themselves to continue the projects they started. We have established contact with the award-winning team and are confident that it will result in a publication. In addition, another group, whose project was highly competitive and nearly as good as the awarded one, has expressed ambitious plans and eagerness to collaborate. It is highly likely that this project will also lead to a publication, perhaps even more than one, given its breadth.
A good idea to monitor the execution of the Institute was to run short online surveys on Google Forms (anonymously). The participants were eager to answer them which helped to respond to all the problems as they occurred as well as evaluate our work throughout the whole event.
According to the four surveys executed during the Institute, all presentations given by invited speakers enjoyed high or very high scores from participants. A significant number of participants (83%) responded that the schedule of the classes on a daily basis was well planned. A significant portion of participants (83%) strongly felt that their computer capabilities increased during school. A significant number of participants (80%), early on, said that the methods and skills they learned during the school would be useful in their future work. A very promising number of participants (65%) responded that they enjoyed working on the group project, and even more of them (94%) said they definitely intend to further explore computational methods in their work. All surveyed participants declared that if they had to decide to participate in the school once again - they would do so. We have also received a couple of heart-warming messages in the final survey:
We accepted with gratitude the feedback from participants who kept saying that there were two best things about the Institute: people (open-minded, enthusiastic and cooperative) and the possibility to grow (learning new research methods, and programming). We note that this was possible not only thanks to the support and funding from various institutions and thanks to highly-involved participants. We believe that this was achieved also by a committed group of organizers who looked out for each other and worked as one team, often beyond “working hours”.
Outreach and application process The outreach and application process can be characterised as smooth overall. We have used the SICCS webpage, the website of the Computational Social Science Center at the University of Bologna, Twitter, Bluesky, PolMeth and EPSA email lists to advertise the Summer Institute. Our idea was to fix the schedule relatively early to convey a firm idea of what to expect. One problem from our view was that we received notification about funding relatively late (February), which complicated the invitation of speakers for June. Nevertheless, we could still put together a programme relatively quickly that gave applicants a precise idea of the Summer Institute.
We have received an exceptionally high number of applications compared to previous editions. There were around 50 applications for 15 places. Our selection process was guided based on three criteria (a CV, a statement of interest, and a writing sample). Moreover, we have considered the career stage, field of study, and familiarity with methods to assemble a balanced group. Three organisers thoroughly reviewed all applications and decided on the 15 highest-ranking individuals.
Creating a rank list of students allowed us to create a waiting list to invite more students in case accepted applicants jump off. Indeed, this happened in two cases, and since we already knew who to invite next, we did not lose any seats despite receiving this notice relatively late.
Due to the great interest in the Summer Institute, we have decided to offer people who were not accepted to attend the activities during the complete first week online as well as the keynote speakers in the second week. As our Summer Institute was mostly organised hybrid already, offering this option didn’t require additional preparation or costs. While many people initially expressed interest in taking up this opportunity, we ended up with approximately five people who joined consistently during the Summer Institute. However, they were extremely positive about the possibility of online participation and their learning curve. This makes us optimistic about offering this option on future occasions.
Upon arrival, we strongly suggested that accepted participants complete the SICSS boot camp. Moreover, we have created a syllabus that aligns with the schedule to enable students to prepare for the Summer Institute. We have shared this relatively early to put students in the position to prepare themselves. As the first week was held online, we have further attempted to clarify the technical requirements for joining remotely early. Slack has proved difficult here as every student can join easily, but only after repeated technical assistance. We have shared the “SICSS On-boarding Form” to add students to the network. Lastly, we asked students to indicate their dietary requirements for social dinner and coffee breaks.
We have received a few questions from students during this period. Most of them asked where the Summer Institute takes place and where best to book accommodation. Therefore, in the future, it might be helpful to send students an information package providing them with the location of the department and suggestions for accommodation. Originally, we shared a map of Bologna with some additional information relatively shortly before the start.
The first week took place on June 11 until June 14. These four days online were designed to bring students the same level of methodological knowledge of the basic toolbox of computational science and to familiarise them with current debates in the field. Knowing that it is generally difficult to convey a lot of content online, we have kept the days relatively short. Each day, there was a workshop that taught methods and a presentation by scholars introducing larger conceptual issues.
On three out of four days, we offered optional assignments, which allowed students to exercise the methods taught in the workshops at home. The idea was not to make this mandatory because not everyone would have time to submit them. The tasks were kept relatively simple to give less experienced students, in particular, the opportunity to improve before the group work in the second week of the summer institute. Additionally, we offered online office hours in the afternoon where students could consult us on the assignments or the material discussed in the Summer Institute. While few students completed the assignments, none attended the online office hours.
A fundamental question that had to be weighed up here was whether students should work on their individual research questions (such as their doctoral thesis or master’s thesis) or in groups during the second week. We decided in favour of the latter because it was our impression from previous versions of SICSS that the learning dynamic would otherwise be too individual. Group work allows students to participate equally in the content of the Summer Institute.
To facilitate the group work to take place in the second week, we have already created the groups based on students’ level of experience to create balanced groups. Thus, the first week concluded with creating break-out groups where the groups could already get to know each other and agree upon a topic before arriving in Bologna. This also allowed students to communicate with each other if they had arrived earlier.
The second week took place between June 17 and 21. Before the start of the Summer Institute, we created name tags and a welcome package (notepad, jersey/t-shirt, etc.), which we allocated to each student at the beginning of the first week. The second week was designed according to the same principle when we have a keynote speech in the morning, introducing a ‘bigger’ theme of CSS, and group work in the afternoon. Where it was possible, keynote speakers stayed throughout the afternoon to answer questions and help students with their projects.
This format has turned out very favourably. As most of the presenters spent the whole day with us (some even over two days), there was a very friendly atmosphere between organisers, presenters and participants, which characterised the atmosphere of the whole Summer Institute. This has contributed to a productive learning environment where individual needs were addressed.
Students could work in the department until 6 pm and consult the organisers for any questions. However, overall, participants were rather shy about consulting us for questions. In the future, it might thus be helpful to organise a mandatory office hour/meeting with students in the middle of the week to encourage them to take stock of the advancements of their group project.
A more specific problem was that students asked for more computing power for their project. This has made us aware that we should define a limit to the group work to be achieved during the week. Certainly, for various reasons, it is not necessary to come up with a computationally intensive project; rather, it is necessary to present an outcome that tests the research question on a smaller sample of the data. This should be emphasised more clearly in the future.
The Summer Institute concluded with the presentation of the group projects on the last day. We have invited faculty members to attend students’ presentations so that participants receive valuable feedback on their work. Looking at the varying quality of the presentations, the question of whether students should work on individual projects or groups re-emerged. In view of the different progress made by the groups, it was difficult overall to provide substantial feedback at the presentations.
After the Summer Institute concluded, we issued a certificate of attendance for each student. This has been particularly useful for students who asked for funding from their institutes, which requires them to provide formal proof of participation. Thus, we have created a template for future occasions.
We received a lot of extremely positive feedback from the participants. From our point of view, the most pleasing feedback was that both participants with less experience and advanced participants expressed that they found the event useful. As the imbalance between advanced and less advanced students could have aggravated the learning process, we feel vindicated in the design of the programme and the selection of participants.
Furthermore, it was interesting for future events that students explicitly stated that they prefer the format with group work to the format with their individual projects. In this respect, we were very pleased to hear that one group has even started to turn the SICSS project into a publication.
We have thought about creating a survey that specifically asks about the experience in order to improve the programme in the future with this feedback. However, we felt students already received a lot of emails during the two weeks of the Summer Institute. Therefore, we decided that limiting ourselves to considering the feedback received in face-to-face conversations would be more beneficial.