For *ACL workshop and LREC-COLING there are now templates to follow. Please use these as it makes it easier to ensure that all information has been filled in when submitting a proposal, and for the reviewers it makes it easier to compare submissions with one another. Please refer to the respective calls for workshops to find the templates.
Note that when you submit a workshop proposal, you can order your preference in which conferences you wish to be colocated with. This however, is simply a request and you should not expect that it is necessarily held. However, there may be particular reasons that your workshop cannot be colocated with a conference, and you should specify this. Without this information, you may have your workshop assigned to a conference that you are unable to host a workshop at (for whatever reason).
Simply stating that the reviewing pool is diverse is not enough. When we are talking about diversity, we do not mean diversity between academia and industry. In particular when talking about diversity, we mean that there is a focus on diversity in the invited speakers (past and planned), the organising committee, the fields that are covered e.g. also having social scientists or linguists submit. This is in addition to diversity between academia and industry and the reviewing pool. The point here is that simply highlighting on area does not cover what we understand by diversity. Certain workshop topics are inherently diversity focused which should be highlighted. For instance, when workshops are addressing questions such as low-resource languages or creating a space for different fields/disciplines to meet, the topic of the workshop itself is focused on a form of diversity.
If you are only able to show diversity along parameters of academia & industry and seniority, you’re a step behind workshops that take the diversity question seriously. The point here isn’t that industry/academia diversity isn’t important, simply that in my opinion, it is not at all enough compared to what many workshops are doing now.
Having said that do highlight topics of diversity that are inherent to the topic or focus of the workshop (that year). e.g., A workshop on summarisation that is focusing on summarisation and simplification for people with reading comprehension difficulties should absolutely highlight that the topic of the workshop is intrinsically linked to diversity in the field (back this up by inviting experts to speak on the topic).
The description of each workshop organizer should seek to answer two questions: 1) How does the organizer’s research experience relate to the workshop (e.g., listing areas of research) and 2) what organizational experience does an organizer have (e.g., academic service, organizing academic events, public events such as NLP meetups, etc.).
An organizer’s academic history (i.e., where they got their degrees or where they have held positions) is less relevant information.
Please, for the love of God, provide short bios of the invited speakers. This allows the reviewing committee to evaluate whether the invited speakers indeed are a good match or not. This is actually important.
Provide accurate information. This should go without saying, but give the number of submissions that were received in previous years (this is an evaluation criteria but the evaluators are the community). Tell us how many people were physically in the room (this is not an evaluation criteria, but it is important to allocate appropriately sized rooms for workshops).
Workshops are not a popularity contest. Workshops focusing on genuinely far-from-the-norm topics are evaluated equally with other workshops, whether 5 or 50 submissions were received. But if you have a workshop on a topic that sees a steady decrease in submissions, or have a workshop that focuses on a popular topic but very few submissions, then I would personally count that against the workshop (I.e., do some work to address this for instance by expanding the scope or having a particular theme of the workshop). But please provide accurate information, without which there is no basis for evaluation and no basis always provides room for negative evaluation.
Some years, there are not a lot of spaces for workshops and it is the smallest things that make the difference. So for that reason, I would recommend not describing things that are not settled. For instance, if you are planning a shared task, but do not have a chair to organise the shared task, it would be better simply not to list it. The exception to this rule are speakers and panels: Do note who you have reached out to, whether they have responded or not.
One thing that is highly considered is whether a workshop is bringing a new perspective - this goes for new and recurring workshops in slightly different ways.
New Workshops: For new workshops, an emphasis of the proposal should be what topics/space the workshop offers that hasn’t been addressed thus far.
Recurring Workshops: For recurring workshops, they should emphasise how they innovate over previous iterations. This can be in terms of the theme of the workshop, the topics addressed, etc.
For new workshops, finding out exactly what is being looked for in workshop proposals, and in my experience it is useful to focus on what gaps the workshop seeks to close. If there are competing workshops, then what isn’t being offered in them or why it’s necessary with multiple workshops in the area (i.e., the area is so popular/has so many nuances that a single workshop isn’t enough).
When you’re at this point you should know how to write a workshop proposal, but here are helpful areas to focus on:
Organising a workshop can be roughly split into N large tasks:
This step should be taken early and reviewers are generally a trustworthy bunch in terms of getting reviews in, but collecting a large enough program committee that can address all of the submissions can be a time consuming tasks, so it should be started early, and there should be plans for emergency reviewers.
Typically between 14-30 days is enough time for reviewing. It is useful to ask some reviewers to be emergency reviewers designated before the review process begins, as there are always reviewers who do not finish their reviews and you will therefore need to do it.
Obtaining submissions is in large effects a matter of how much the community is aware of the workshop. To this end, make sure to announce the workshop and submission dates are publicised across social media (Twitter being the primary one) and mailing list.
An incomplete list of mailing lists (please e-mail me if any list is missing): Corpora, Linglist (AKA Linguist List), ACL Mailing list (automated after making an event on the ACL portal), SIG* (many of the special interest groups have mailing lists that it’s worth posting in). For more social science-y workshops it can also make sense to send a call to the Association of Internet Research (AOIR) mailing list.
Make sure to send out several calls, and highlight any changed information (e.g., deadline extensions).
It is typically a good idea to give authors ~10 days to address reviews. Decisions are also useful to make on the basis of whether the issues raised in the review can be addressed in that span of time. After authors have uploaded their camera ready versions, you should check them to make sure that they adhere to the ACL style. If they do not, help the authors address the issues for compliance.
The folks who are running the virtual platform will need to know the submission ID for each paper that you accept. This will typically you will be requested the following information in a spreadsheet for each paper:
For the Gathertown and Zoom questions, only one link will be produced for each but do answer in all rows, to make sure that all papers are correctly assigned by the virtual conference platform.
The virtual conference provider will set a deadline for uploading the content. This is a hard deadline that must be taken into account when planning the workshop schedule.
Recommendation
I would recommend that you have every presenter create a poster and a video presentation. This is more work for the authors before the conference, but it means that both talks and posters are treated equally after the workshop has concluded.
It is important that workshop organizers know one single thing: Any delay from one workshop delays the proceedings for the main conference and all other workshops. For this particular thing, delays are unacceptable.
I have a preferance for softconf, but regardless of choice, make sure to add the workshop chairs as admins of the workshop submission site. This will help if there are any time-sensitive issues.
We use ACLPUB2. Using ACLPUB that is built into softconf has been deprecated. While ACLPUB2 is still very new and currently in active development and has some bugs, this is the process that will be used moving forward. Do report bugs as you find them, and ideally, create pull requests for any bugs that you fix.
program.ymlpapers.yml.description.yml.organizing_committee.ymlThe only date format that is accepted is: YYYY-MM-DD HH:MM:SS
A common source of errors in the proceedings is missing or incorrect syntax. For instance, I spent an entire day only to find out that in program.yml, if you specify the paper IDs that go into each session, then the format is:
papers:
- id: <ID>
NOT
papers:
id: <ID>
And sadly, the error messages are not always super helpful.
I would use the typical debugging process:
If a bug appears, or something could be improved, submit a bug report and create pull requests once you’ve fixed it.
program.ymlShould contain:
papers.yml.papers.ymlTasks:
papers.yml reflects the author lists in the PDF. A reasonable person would expect that authors would make sure that their own information was correct. A reasonable person would also be wrong.All findings papers presented at the workshop need to be added to papers.yml to be included.
Recommendation: As all papers in papers.yml need to have an ID, I would recommend giving each finding paper alphanumeric IDs, starting with F1 to F[n].
description.ymlShould contain:
workshop_<workshop_[number]> (e.g., workshop_1)prefaces.ymlThe format of the file is:
- title: <Section Heading>
- file: <Path to file>
The title will be used as the section heading in the compiled proceedings, so if you would a sectioned titled “Introduction” that is defined in introduction.tex located in the prefaces/ directory. Your prefaces.yml should look like this:
- title: "Introduction"
- file: "prefaces/introduction.tex"
All files will be included in the proceedings.tex file, so there is no need for begin/end document or a preamble. Simply have the content in the file.
softconf directory over to the workshop repository.softconf2aclpub.py.archival: false to all papers that are non-archivalattachments where the attachment is <submission number>.txt.F<number>.id are strings.organizing_committee.yml is generated with the organizers’ information.program_committee.yml.\x).description.yml file.Clone the ACLPUB2 repository and the workshop Github repository into the same root directory. The structure should look like this:
├── root_folder (e.g., ~/Documents)
| ├── aclpub2
| ├── workshop_repo
Create a virtual environment for building the proceedings and installing python packages. Personally, I use virtualenv, so my instructions are for that but use any you like.
Go to the aclpub2 directory (and subdirectories) and install all packages.
Add: export PYTHONPATH="/path/to/aclpub2" to your virtual environment definition file (e.g., for virtualenv, the file is venv_dir/bin/activate)
Sometimes, the proceedings scripts or softconf2aclpub.py will fail because of different field names, this should be manually addressed in the appropriate tool.
Steps:
/path/to/aclpub2/bin/generate . --proceedings --overwriteACLPubCheck is a tool is developed to identify issues with the submitted camera ready PDFs once the proceedings have compiled. What this tool checks for is things like formatting issues and font issues. This means that it is run after the proceedings have been compiled and compilation issues will therefore not be caught by it. Moreover, this does not capture issues like: Missing metadata (e.g., not all authors being listed in the YAML file), missing papers in the proceedings PDF, or accidentally added papers in the proceedings PDF (e.g., Findings papers that are accidentally included in the proceedings rather than just the program). This is a super helpful tool and the person building the proceedings absolutely should use it.
Each workshop will get a Github repository where all files (PDFs and metadata) are to be uploaded to. These repositories will be created by the conference and each workshop will be provided access to the github users that they provide to the workshop chairs for their conference. An example repository will be made available for viewing soon.
Once all YAML files have been created, you can then pull out the information needed Underline (with the exception of type of talk and paper ID) using the following script (courtesy of Rob van der Goot) to add to the spreadsheet:
data = yaml.safe_load(open('WNUT_data.yml'))
for paper in data:
abstract = paper['abstract']
title = paper['title']
output = [title, abstract]
for author in paper['authors']:
first_name = author['first_name']
second_name = author['last_name']
email = ' ' if 'emails' not in author else author['emails']
output.extend([first_name, second_name, email])
print('\t'.join(output))
The very first bit of advice that I can offer is: Create a google group for all communication with workshop chairs. It streamlines information a lot and it makes it a lot easier. I cannot recommend this enough.
There are some deadlines that are soft and others are hard. It is usually helpful to workshop organizers if the proceedings & video upload deadlines are pushed back, since these are often what influences submission and camera ready deadlines.
The information needed to generate ISBNs is incredibly simple: The title of the proceedings for each workshop. This can be obtained quite early on, so that once main conference proceedings ISBNs are ready to be requested, the workshop ones can be ready. I would suggest creating a spreadsheet early and sending it out, the titles of proceedings should be checked against this spreadsheet.
Developing the proceedings and dealing with workshops can be a time-consuming and error prone task. Make sure to suggest that workshop chairs dedicate time at least 1 week in advanced of submission. This also means that the camera ready deadline needs to be at least 8 days before the proceedings deadline. Proceedings cannot be generated until workshop chairs create the github repositories, so create these early and give access to all workshop chairs.