⬆ Design Documents Overview

DocNow Quotas

creator: Ed Summers
created: 2019-07-25


The DocNow application currently allows users to log in and start up data collection from Twitter and let it run for as long as they want. This is by design, since it should allow the long term collection of hashtags for example. However we’ve had some users come into the application and start data collection for a trending (high-volume) hashtag, and then never turning it off. In some cases this has resulted in millions of tweets being collected, which have filled up our ElasticSearch database, which negatively impacted other users.

While we would like the application to support high volume data collection this shouldn’t be available for anyone who logs in. This is especially the case for the demo.docnow.io instance which is running really just to show users how the app works, with the idea that they would install their own instance if they wanted it to use it “for real”.


In Scope:

Out of Scope:


When installing the DocNow application the administrator will see an input box on the Settings page that sets a tweets-per-user limit for users of the application. The default will be set to 50,000 but can be changed to an arbitrary number, or left blank if the admin would like there to be no limit. In addition the admin can visit a page for individual users where they can see how many tweets the user has collected, a list of their collections, and the ability to set the tweets-per-user limit for only that user.

When a user collects more than the quota-limit all data collection jobs by the user will automatically stop. When the user attempts to restart data collection or explore more data they will get an error modal screen that explains that they have reached the limit, and be asked to delete some of their collections if they want to continue doing data collection. The user will not be prevented from interacting witht he data they have already collected.

Action Items