BSides Orlando is excited to have completed its first blind CFP. To help increase transparency about our talk selection process and encourage feedback from the community we are publishing our process for selection, outlining board selected slots, and discussing areas we noted for future improvement. We hope this disclosure helps us continue our path of positive changes that make for a better BSides Orlando.
CFP Process
This year we completed a blind CFP process. At first, we attempted to use OpenCFP. During our testing, we were not able to achieve an outcome that made us feel comfortable using it for selection. We hope to revisit this platform and others next year.
We continued with the previous Google Forms process for CFP entries. A small number of board members and staff were able to see the full CFP details including the name and bios. Once the CFP closed, those board and staff members migrated the talk titles and abstracts to a separate voting sheet. We sought to redact identifying information as much as possible to anonymize the speakers during voting. Each speaker was given a numeric ID. Some speakers submitted multiple talks. One of the goals was to make sure we didn’t end up selecting eight talks from two speakers. If a speaker had two or more talks selected in the blind process, the talk with the most votes would be selected.
Staff members then reviewed the content of the abstracts for talks we felt best fit the needs of the conferences, and showed the most benefit as a talk, and then voted on those based on merit.
Voting
Voting was a simple process where each staff member and volunteer on the selection committee rated each talk in the spreadsheet without speaker names or bios. The designation yes, no, or maybe was used for voting. A calculation then gave a weighted response for the number of yes and maybe votes. The top seven talks from the weighted voting received acceptance letters.
After voting was complete the bios and speaker names were revealed to the voters.
Talk Slots
BSides Orlando 2018 is a one-day event running two tracks of talks for a total of ten talk slots. The staff and board reserved three talk slots leaving seven open for the CFP. The three reserved slots are the opening and closing keynotes and one technical slot that ensured, even after a blind CFP review, we would have at least one strong penetration testing talk from seasoned professionals. By reserving these three slots we could provide seven opportunities for new speakers to get a slot based on merit, via the blind CFP, while still guaranteeing at least three talks from industry professionals that could help expand our audience’s professional knowledge. We felt this was an excellent way to encourage new voices while making sure the audience got to hear from industry pros.
Demographics
Below are some demographics about the submitted talks and the review committee.
Submissions
- 24 for talks submitted from 18 speakers
- We do not have data on the gender of submitters
- We can make assumptions based on common first name gender association. However, some speakers submitted talks under handles.
- Using common assumptions we believe all submitters are male.
- We do not have any statistics on race or nationality as this is not data we gather during the submission process.
Reviewers
- 57.1% female and 42.9% male
- Two board members, three staff, and two volunteers
Improvements
With any process there is always room for improvement. Here are some areas we identified during the process where we would like to grow:
- Start CFP earlier
- We wanted to wait until the venue is finalized to open the CFP. In the future we will open the CFP earlier realizing the venue will be in the metro Orlando area and a specific venue is not necessary to open the CFP.
- Increased CFP promotion
- We are looking for ways to reach out to different groups that will help us bring new security voices to the stage. We would like to partner with groups that can help us bring new and underrepresented voices to the BSides Orlando stage. If you would like to join us to form a BSides Orlando outreach team, please contact us!
- By using a blind CFP process we can help ensure we have more talks selected by merit. We hope this will help create a more diverse speaker set.
- CFP Technology
- Google Forms served its purpose for our CFPs in the past, but we are looking for something with more modern and customizable features.
- Specifically, built-in voting and blind review options are very important to us.
- The blind CFP review process we used this year was a great step forward. However, we would like to have something that makes calculation and voting less difficult while being more automated.
- Reviewers
- Eventually we would like to develop a team of industry professionals that are not directly affiliated with the conference to perform the CFP review.
- By removing the staff and volunteers we create a selection scenario with less bias.
- A review panel independent from the BSides Orlando organization can remain free from the pressures of running the conference and focus strictly on the talks and their merit.
- Review would still remain blind, but independent experts and industry pros, with their perspectives on security trends, may be better able to spot talks that deserve exposure.
- Statistics
- We may look at gathering more statistics about CFP submissions to help us evaluate how effective our outreach programs are in the future.
- This would be purely voluntary on the part of the submitter.
- First Time Speakers
- We would like to create a first time speaker program.
- Submitters could indicate they are a first time speaker.
- This would put them in a different voting pool where they would not be in direct competition with more experienced speakers.
We hope you enjoyed this new level of transparency into our process. We continue to strive to be a better conference year over year. This is the next step BSides Orlando has taken to make a better event for our audience.