This case study was written as a collaboration between Drupal Association staff and Technology Supporting Partner Distil Networks.
Drupal.org is the home of one of the largest open source communities in the world. We've been online for more than 13 years and collectively we build the Drupal software, provide support, write documentation, share networking opportunities, and more. The open source spirit pushes the Drupal project forward, and new members are always welcome. It falls to us to maintain our community home and preserve the welcoming atmosphere that leads people to say,"Come for the code, stay for the community."
As stewards of Drupal.org, it's our responsibility to give the community a voice and welcome everyone who wants to participate in the project. At the same time, there are bad actors who would take advantage of our open community and platform for abusive purposes.
Drupal.org long-standing presence on the web has given it authority in the eyes of search engines. The site hosts millions of pages of content - all generated by our users. This combination of authority and open access for users to create content makes us a very high value target for phishers and spammers.
Spam is a nuisance to our existing community, devalues our project to the newcomers we are hoping to welcome, and left unchecked could degrade our search presence.Challenges Spammers create bogus accounts to post their junk content
Only registered members can post content to the Drupal.org website, so there's a continuous onslaught of actors attempting to create accounts for the purpose of inserting link spam and other bad content onto the site. In the past, we've implemented a variety of strategies such as content analysis, behavioral analysis, social moderation, and rate limiting. And while these measures have been effective at reducing some of the spam we've seen, the onslaught continues.
The reason for that? Much of our attempted spam is not coming from bots. These are real people using tools to cloak their identity and manually creating accounts en masse. In many cases they may not even post junk content immediately. They will often sit on "sleeper" accounts waiting to be paid by somebody to promote malicious content.It's too time consuming to manually remove spam content
Spam fighting is also a thankless task. All time spent fighting spam, whether by members of the engineering staff or our incredibly dedicated community volunteers, is time not spent on the project. Spam fighting has an opportunity cost that creates burn-out among staff and volunteers, and is not something we can afford to leave to manual moderation.
Especially when it comes to our community volunteers– they want to spend their time helping people with Drupal technical questions, not deleting spam.Fake accounts and spam pollute the community engagement metrics
There are 1.9 million user accounts in the Drupal.org database, but using this data to measure community engagement is challenging because of the number of spammer accounts that have been registered over the years. When we have to work around so many illegitimate accounts, it's difficult to determine metrics for community health such as if our legitimate user growth is increasing or decreasing. We need cleaner user account data to give us more reliable community metrics, and help us make informed decisions.The Solution
Before reaching out to Distil Networks, Drupal.org relied primarily on two modules to help us fight spam. Mollom is a Drupal stand-by—a content analysis tool that looks at what users are posting and compares them against known bad actor patterns. This content analysis helps us identify and block new waves of spam patterns, but it doesn't prevent these waves from being posted in the first place.
The second module we use is Honeypot, which uses a combination of honeypot and rate limiting methods to prevent bot spam. Honeypot does a good job in preventing mass spam attacks by bots, but when real people are creating the underlying accounts honeypot can't help us.
As we researched ways to prevent spam, we discovered that all of these bad actors we wanted to keep out had one thing in common—they are hiding their identities behind proxies. This prevents us from simply blacklisting certain ip addresses or ranges. So instead, we began researching ways to unmask the users behind these proxies and block them before they can even create an account.
Our research led us to Distil Networks. We now run the Drupal.org registration pages(and only the registration pages) through the Distil Cloud CDN. Distil's service gathers device fingerprints for the users trying to create the accounts, and we're able to leverage those fingerprints to block users who would otherwise generate dozens or hundreds of accounts by rotating through proxies. This fingerprinting process is limited to a hashed, unique identifier and only affects our registration process, to preserve the privacy of our legitimate users.What the Distil data shows
After enabling Distil's service for our registration process we were able to capture fingerprints for about 20,000 account registrations over the course of nine months. We were immediately able to identify more than 10% of those account registrations as duplicate registrations by the same user, hiding behind a proxy. As we dug into the data further, we realized that thousands of the spam accounts that spammers are attempting to register are actually created by only 200-300 real individuals.
By blocking these 200-300 individuals by their Distil fingerprint, we can block thousands of account registrations, and tens of thousands of spam posts that would have been created had these 'sleeper accounts' been activated.Results
Even with Distil's sophisticated profiling tool available to us, we knew that the spam fighting process would continue to have a manual component. In the first place, there are still thousands of 'sleeper' accounts registered before we implemented Distil that could be activated. And secondly, we know that we cannot simply rely on proxy detection and fingerprint collisions to identify spam accounts. Some of our users are in countries where a proxy is the only way to access a free and open internet. Other users are in environments that have identical device fingerprints and a shared IP, such as a classroom computer lab.
However, by taking advantage of the tools that Distil offers, we can now stop many of the account registrations at the source. In the same time that it once took us to moderate a single new user account that had just posted spam, we can now block a unique id that would have been used to create a dozen or even a hundred more accounts.
We've seen trends in our account registration logs that show that the new methods are working. As we block spammers in ways they can't circumvent through proxies, their ability to register multiple accounts diminishes. Without being able to mass register accounts to later activate when selling link spam, Drupal.org becomes a less viable target.
While some spam still gets through, whether from old sleeper accounts, or lucky new spammers that manage to slip by, the overall reduction in spam has been significant. This lets our volunteers and internal staff direct more of their efforts at moving the project forward, rather than fighting spam.
With fewer illegitimate account registrations, we're also able to improve the metrics we use to measure our community health and engagement, by lowering the noise-to-signal ratio in user activity.Conclusion
We want to thank Distil Networks for joining the Drupal Association as a Premium Technology Supporter. The tools that Distil Networks provide enable us to better take care of the home of the community. Fighting spam is a never ending challenge: as long as there is a financial incentive to posting spam, bad actors will continue to evolve their methods, but with a partner like Distil Networks we are now equipped to stay one step ahead.
To learn more about how Drupal.org and Distil Networks partnered to tackle spam, and to learn how you could leverage a similar solution for your own site, please join us at our webinar on April 5th, at 10am Pacific.
Distil Networks will be joining us at DrupalCon Baltimore from April 24-28th. We invite the community to join us there and learn more about our partnership.
Any user on Drupal.org who has accepted our Git usage policy may now create full projects with releases. This is a big change in policy for the Drupal project, representing an evolution of the contribution ecosystem in the past half a decade.What was the Project Application Process?
Ever since the days when Drupal's code was hosted in CVS there has been some form of project application process in the Drupal Community. To prevent duplicate, low-quality, insecure, or otherwise undesirable projects from flooding Drupal, users would submit sandbox projects to an application queue to be reviewed by a group of volunteers.
After resolving any issues raised in this review process, the user would be given the git vetted role, allowing them to promote their sandbox to a full project, claim a namespace, and create releases. Once a user had been vetted for their first project, they would remain vetted and be able to promote any future projects on their own, without submitting an additional application.The Problem
Unfortunately, though the project application process was created with the best of intentions, in the long term it proved not to be sustainable. Drupal grew too fast for a group of volunteer reviewers to keep up with reviewing new projects, and at times there were applications waiting in queue for 6 months to 1 year, or even more. That is much too slow in the world of software development.
This put Drupal in a difficult situation. After years of subjecting new projects and contributors to a rigorous standard of peer review, Drupal has a well-deserved reputation for code quality and security. Unlike many open source projects, we largely avoided the problem of having many duplicate modules that exist to serve the same purpose. We unified our community’s effort, and kept up a culture of collaboration and peer review. At the same time, many would-be contributors were unable or unwilling to navigate the application process and so simply chose not to contribute.
The question became, how could we preserve the emphasis on quality while at the same time removing the barrier to contribution that the application process had become?Constraints on a solution
Opening the contribution gates while retaining strong signals about code quality and security was a tricky problem. We established three constraints on a solution:
- We need to welcome new contributors, and eliminate the walls that prevent contribution.
- We need to continue to send strong signals about security coverage to users evaluating whether to use modules from Drupal.org.
- We need to continue our strong emphasis on quality and collaboration through changes to project discovery that will provide new signals about code quality, and by providing incentives and credit for peer review.
In collaboration with the community, the security team, members of the board, and staff we outlined a solution in four phases:Phase 1: Send strong signals about security advisory coverage.
- We updated project pages to include messaging and a shield icon to indicate whether a project received security advisory coverage from the security team.
- We now serve security advisory coverage information in the Updates status information provided by Drupal.org, and we're working on a patch to display that information directly on the updates page of users' Drupal sites.
Here are some examples of what these security signals look like on project pages:
If a project is not opted in to security advisory coverage, this message will appear at the top of the project page:
And this one will appear near the download table:
If a project has opted in, this message will appear near the download table:
And covered releases will show the coverage icon (note how the stable 7.x release has coverage and the 8.x release candidate does not):Phase 2: Set up an opt-in process for security advisory coverage
- Previously any project with a stable release would receive security advisory coverage from the security team. As we opened the gates for anyone to promote full projects, the security team needed an opt in process so that they could enforce an extra level of vetting on projects that wish to receive advisory coverage.
- We agreed to repurpose the project application queue to be a queue for vetting users for the ability to opt their projects in to receive security advisory coverage. Now that this process has been decoupled from creating full projects, the security team may revise it in future–in collaboration with staff and the community.
- Now a project maintainer must opt in their project to receive advisory coverage and make a stable release in order to receive security advisory coverage from the security team.
Once a maintainer has been vetted by the security advisory opt in process, they can edit their project and use this field set to opt-in:Phase 3: Open the gate to allow users to create full projects with releases without project applications.
This is the milestone we've just reached!Phase 4: Provide both automated code quality signals, as well as incentives for peer review of projects - and factor these into project discovery
- We are working on this phase of the project in the issue queues, and we appreciate your feedback and ideas!
So in the end - what is the new process if you want to make a contribution by hosting a project on Drupal.org?
- You must have a Drupal.org account, and you must accept the git terms of service.
- You can create a sandbox or a full project
- Note: We still strongly recommend that project maintainers begin with sandbox projects, until they are sure they will be able to commit to supporting the project as a full project, and until the code is nearly ready for an initial release.
- That said, you can promote a sandbox project to a full project at any time, to reserve your name space and begin making releases.
At this point, you will have a full project on Drupal.org, and will be able to make releases that anyone can use on their Drupal site. The project will not receive security advisory coverage, and a warning that the project is not covered will appear on the project page and in the updates information.
If you want to receive security advisory coverage for your project, you will need to take these additional steps:
- You must apply for vetted status in the security advisory coverage queue.
- Members of the security team or other volunteers will review your application - and may suggest changes to your project.
- Once feedback is resolved, you will be granted the vetted role and be able to opt in this project, and any future projects you create, to receive security advisory coverage.
- Note: Only *stable* releases receive security advisory coverage, so even after opting your project in you will not receive the advisory coverage shield except on stable releases.
Now that the project application process is no more, the gates are open. We are already seeing an uptick in projects created on Drupal.org, and have seen some projects that had migrated to other places (like GitHub) migrate back to Drupal.org. We can expect to see contributions from some great developers who previously felt gate-kept out of the community. We will also see an uptick in contributions that need work, from new developers and others who are still learning Drupal best practices.
That is why our next focus will be on providing good code quality signals for projects on Drupal.org. We want to provide both automated signals of code quality, and new incentives for peer review from existing members of the community. We're outlining that plan in the issue queues, and we welcome your feedback and contributions.
We also still have work to do to communicate this well. This is a big change for the Drupal community and so we want to make people aware of this change in every channel that we can.
Finally, after such a significant change, we're going to need to monitor the contrib ecosystem closely. We're going to learn a lot about the project in the next several months, and it's likely there will be additional follow ups and other changes that we'll need to make.Special Thanks
There are many, many contributors on Drupal.org who have put in time and effort to help make the contribution process better for new contributors to Drupal - the deepest thanks to all of you for your insight and feedback. We'd also like to specifically thank those who participated in the Project Application Revamp, including:
- Donna Benjamin of the Drupal Association Board
- Dries Buytaert founder of Drupal
- Angie Byron of the Drupal Security Team
- cilefen of the Drupal Security Team
- Mark Ferree
- Michael Forbes
- Gisle Hannemyr
- Michael Wayne Harris
- David Hernandez
- Michael Hess of the Drupal Security Team
- Michelle Jackson
- Greg Knaddison of the Drupal Security Team
- Damien McKenna of the Drupal Security Team
- Benjamin Melançon
- Emilie Nouveau
- Alberto Paderno
- Håvard Pedersen
- Andrii Podanenko
- Alex Pott of the Drupal Security Team
- Klaus Purer of the Drupal Security Team
- Steve Purkiss
- David Rothstein of the Drupal Security Team
- Roy Scholten
- Mark Shropshire
- Jeremy Thorson, who first proposed the revamp as a community initiative
- Jordan White
- Peter Wolanin of the Drupal Security Team
- xjm of the Drupal Security Team
- and many others.
Drupal 8.2.7, a maintenance release which contains fixes for security vulnerabilities, is now available for download.Download Drupal 8.2.7
Upgrading your existing Drupal 8 sites is strongly recommended. There are no new features nor non-security-related bug fixes in this release. See the 8.2.7 release notes for details on important changes and known issues affecting this release. Read on for details of the security vulnerabilities that were fixed in this release.
- Advisory ID: DRUPAL-SA-CORE-2017-001
- Project: Drupal core
- Version: 8.x
- Date: 2017-March-15
When adding a private file via a configured text editor (like CKEditor), the editor will not correctly check access for the file being attached, resulting in an access bypass.Some admin paths were not protected with a CSRF token - Drupal 8 - Cross Site Request Forgery - Moderately Critical - CVE-2017-6379
Some administrative paths did not include protection for CSRF. This would allow an attacker to disable some blocks on a site. This issue is mitigated by the fact that users would have to know the block ID.Remote code execution - Drupal 8 - Remote code execution - Moderately Critical - CVE-2017-6381
A 3rd party development library including with Drupal 8 development dependencies is vulnerable to remote code execution.
This is mitigated by the default .htaccess protection against PHP execution, and the fact that Composer development dependencies aren't normal installed.
You might be vulnerable to this if you are running a version of Drupal before 8.2.2. To be sure you aren’t vulnerable, you can remove the /vendor/phpunit directory from the site root of your production deployments.Solution
Upgrade to Drupal 8.2.7Reported by Editor module incorrectly checks access to inline private files - Drupal 8 - Access Bypass - Critical - CVE-2017-6377
- László Csécsy
- Wim Leers
- Alex Pott of the Drupal Security Team
- Klaus Purer of the Drupal Security Team
The Drupal security team can be reached at security at drupal.org or via the contact form at https://www.drupal.org/contact.
Follow the Drupal Security Team on Twitter at https://twitter.com/drupalsecurity
One of the key reasons that Drupal has been successful is because we always made big, forward-looking changes. As a result, Drupal is one of very few CMSes that has stayed relevant for 15+ years. The downside is that with every major release of Drupal, we've gone through a lot of pain adjusting to these changes. The learning curve and difficult upgrade path from one major version of Drupal to the next (e.g. from Drupal 7 to Drupal 8) has also held back Drupal's momentum. In an ideal world, we'd be able to innovate fast yet provide a smooth learning curve and upgrade path from Drupal 8 to Drupal 9. We believe we've found a way to do both!Upgrading from Drupal 8.2 to Drupal 8.3
Before we can talk about the upgrade path to Drupal 9, it's important to understand how we do releases in Drupal 8. With the release of Drupal 8, we moved Drupal core to use a continuous innovation model. Rather than having to wait for years to get new features, users now get sizeable advances in functionality every six months. Furthermore, we committed to providing a smooth upgrade for modules, themes, and distributions from one six-month release to the next.
This new approach is starting to work really well. With the 8.1 and 8.2 updates behind us and 8.3 close to release, we have added some stable improvements like BigPipe and a new status report page, as well as experimental improvements for outside-in, workflows, layouts, and more. We also plan to add important media improvements in 8.4.
Most importantly, upgrading from 8.2 to 8.3 for these new features is not much more complicated than simply updating for a bugfix or security release.Upgrading from Drupal 8 to Drupal 9
After a lot of discussion among the Drupal core committers and developers, and studying projects like Symfony, we believe that the advantages of Drupal's minor upgrade model (e.g. from Drupal 8.2 to Drupal 8.3) can be translated to major upgrades (e.g. from Drupal 8 to Drupal 9). We see a way to keep innovating while providing a smooth upgrade path and learning curve from Drupal 8 to Drupal 9.
Here is how we will accomplish this: we will continue to introduce new features and backwards-compatible changes in Drupal 8 releases. In the process, we sometimes have to deprecate the old systems. Instead of removing old systems, we will keep them in place and encourage module maintainers to update to the new systems. This means that modules and custom code will continue to work. The more we innovate, the more deprecated code there will be in Drupal 8. Over time, maintaining backwards compatibility will become increasingly complex. Eventually, we will reach a point where we simply have too much deprecated code in Drupal 8. At that point, we will choose to remove the deprecated systems and release that as Drupal 9.
This means that Drupal 9.0 should be almost identical to the last Drupal 8 release, minus the deprecated code. It means that when modules take advantage of the latest Drupal 8 APIs and avoid using deprecated code, they should work on Drupal 9. Updating from Drupal 8's latest version to Drupal 9.0.0 should be as easy as updating between minor versions of Drupal 8. It also means that Drupal 9 gives us a clean slate to start innovating more rapidly again.
Why would you upgrade to Drupal 9 then? For the great new features in 9.1. No more features will be added to Drupal 8 after Drupal 9.0. Instead, they will go into Drupal 9.1, 9.2, and so on.
To get the most out of this new approach, we need to make two more improvements. We need to change core so that the exact same module can work with Drupal 8 and 9 if the module developer uses the latest APIs. We also need to provide full data migration from Drupal 6, 7 and 8 to any future release. So long as we make these changes before Drupal 9 and contributed or custom modules take advantage of the latest Drupal 8 APIs, up-to-date sites and modules may just begin using 9.0.0 the day it is is released.What does this mean for Drupal 7 users?
If you are one of the more than a million sites successfully running on Drupal 7, you might only have one more big upgrade ahead of you.
If you are planning to migrate directly from Drupal 7 to Drupal 9, you should reconsider that approach. In this new model, it might be more beneficial to upgrade to Drupal 8. Once you’ve migrated your site to Drupal 8, subsequent upgrades will be much simpler.
We have more work to do to complete the Drupal 7 to Drupal 8 data migration, but the first Drupal 8 minor release that fully supports it could be 8.4.0, scheduled to be released in October 2017.What does this mean for Drupal developers?
If you are a module or theme developer, you can continually update to the latest APIs each minor release. Avoid using deprecated code and your module will be compatible with Drupal 9 the day Drupal 9 is released. We have plans to make it easy for developers to identify and update deprecated code.What does this mean for Drupal core contributors?
If you are a Drupal core contributor and want to introduce new improvements in Drupal core, Drupal 8 is the place to do it! With backwards compatibility layers, even pretty big changes are possible in Drupal 8.When will Drupal 9 will be released?
We don't know yet, but it shouldn't matter as much either. Innovative Drupal 8 releases will go out on schedule every six months and upgrading to Drupal 9 should become easy. I don't believe we will release Drupal 9 any time soon; we have plenty of features in the works for Drupal 8. Once we know more, we'll follow up with more details.Thank you
Join us at DrupalCon Baltimore from April 24-28 for a week of inspiration, networking, and learning. Meet Drupal experts and industry leaders who will share new ways to create digital experiences that delight customers, citizens, students, patients, and more.
The event offers programming for decision makers (CIO/Director) as well as digital teams (developers, project managers, site builders, content strategists). Be sure to check out these suggested sessions for both audiences.Top Five Reasons To Attend DrupalCon
- Get inspired! Hear Dries Buytaert’s vision for digital transformation and Drupal.
- Network with peers at 4 industry summits and case study sessions on Bluecross Blueshield, Cornell University, Mass.gov, NBA, Quicken, YMCA, and more.
- Level up your team's skill with 10 trainings and 161 sessions taught by Drupal masters.
- Find solution partners. Visit the exhibit hall to meet Drupal’s robust vendor ecosystem.
- Be Amazed. Meet the open source community that powers Drupal.
Register today. Prices increase March 24th. Attendees can come for the week or just for a day. Plus, the Baltimore Convention Center is easy to reach - just 30 minutes from Baltimore Washington Airport and 15 minutes from the Amtrak Station.
We look forward to seeing you at DrupalCon Baltimore!