DocNow Quotas
Problem
The DocNow application currently allows users to log in and start up data collection from Twitter and let it run for as long as they want. This is by design, since it should allow the long term collection of hashtags for example. However we’ve had some users come into the application and start data collection for a trending (high-volume) hashtag, and then never turning it off. In some cases this has resulted in millions of tweets being collected, which have filled up our ElasticSearch database, which negatively impacted other users.
While we would like the application to support high volume data collection this shouldn’t be available for anyone who logs in. This is especially the case for the demo.docnow.io instance which is running really just to show users how the app works, with the idea that they would install their own instance if they wanted it to use it “for real”.
Requirements
In Scope:
- Limit data collection by user to some number of tweets by default.
- Allow per-user exceptions to the default.
- Allow these limits to be set by the administrator of the instance.
- Provide a way for users to notice that they have reached the limit.
- Provide a way for users to continue data collection after they have hit the limit.
Out of Scope:
- Email notifications about limits.
- In app communication between admin and users.
Proposal
When installing the DocNow application the administrator will see an input box on the Settings page that sets a tweets-per-user limit for users of the application. The default will be set to 50,000 but can be changed to an arbitrary number, or left blank if the admin would like there to be no limit. In addition the admin can visit a page for individual users where they can see how many tweets the user has collected, a list of their collections, and the ability to set the tweets-per-user limit for only that user.
When a user collects more than the quota-limit all data collection jobs by the user will automatically stop. When the user attempts to restart data collection or explore more data they will get an error modal screen that explains that they have reached the limit, and be asked to delete some of their collections if they want to continue doing data collection. The user will not be prevented from interacting witht he data they have already collected.
Action Items
- Implement quota limit input box on settings screen.
- Create an admin page that lists users.
- Create an admin page for individual users that lists their collections and allows a per-user quota limit to be set.
- Add per-user tweet counting during data collection, and the ability to terminate data collection if the limit is met.
- Add modal error screen that explains the limit and how to address, which will get triggered by starting new data collection or exploration after the quota limit has been met.