Documenting the Now develops open source tools and community-centered practices that support the ethical collection, use, and preservation of publicly available content shared on web and social media.
Documenting the Now responds to the public's use of social media for chronicling historically significant events as well as demand from scholars, students, and archivists, among others, seeking a user-friendly means of collecting and preserving this type of digital content. Documenting the Now has a strong commitment to prioritizing ethical practices when working with social media content, especially in terms of collection and long-term preservation. This commitment extends to Twitter's notion of honoring user intent and the rights of content creators. Documenting the Now is a core program of Shift Collective. We are extremely grateful for funding from the Mellon Foundation and technical support from the Princeton University Library.
We build open source tools to help archivists, activists and researchers work with social media data. Follow us on GitHub.
DocNow is a tool for appraising, collecting, and gathering consent for Twitter content. Features include:
twarc is a command line tool and Python library for collecting tweet data from Twitter's official API. It is designed for reliably collecting historical as well as realtime data, and can be used as a software library in your own tools and applications.
Hydrator is a desktop application for turning Tweet ID datasets back into tweet data to use in your research. It has been designed to be a reliable option for researchers who want to use their workstation for long running hydration jobs.
Social Humans is a label system for social media content. SH-C labels are for content creators to share their terms of consent for collection and re-use of their content, beyond Twitter's Terms of Service. SH-A labels are for academics and archivists who are collecting content to share contextual information about the collections. These labels are implemented in the DocNow application (and soon the Catalog), but can be used in any social media archiving program or project, including donor forms.
The Catalog is a community-sourced clearinghouse for tweet identifier datasets. Sharing tweet ids is a practice that is encouraged by Twitter as a way to share research data without negatively affecting users ability to have their data deleted or hidden from the web. We welcome your contributions!
Building and maintaining a community of practice is a core tenet of the Documenting the Now project.
Our online community has more than 600 active members, discussing topics around archiving social media: ethics, technical approaches, research, and practice in the field. Join us on Slack. Follow us at @documentnow on Twitter to follow our work and participate in hosted conversations by following #docnowcommunity.
Documenting the Now runs workshops with activist groups around the country, co-learning and providing training to address activists’ needs and concerns in documenting their work and controlling their narrative. Past workshops have included partners Pettypropolis in Detroit, MI, Texas After Violence in Austin, TX, Don't Shoot PDX in Portland, OR, and the Southern California Library in Los Angeles, CA. If you are interested in conducting a workshop, contact us at firstname.lastname@example.org.
Through our community outreach, we have learned that a primary need of activist communities is support in archiving their work. Archivists Supporting Activists (ASA) is a peer-matching network designed to help activists find archivists that hold themselves to a standard of non-extractive archival practice to support their archival needs. Find an archivist who can help you achieve your goals, or sign up to volunteer your skills.
Documenting the Now has organized three meetings convening multidisciplinary researchers and practitioners to discuss the most pressing issues around social media and web archiving. Most sessions were recorded and are available online:
The best way to follow Documenting the Now news is to follow our Medium blog.
Thanks to our former team members:
Trevor Muñoz, Administrative Lead 2018-2021, Raff Viglianti, Developer 2019-2021, Micha Broadnax, Community Trainer 2018-2020, Jarrett Drake, Community Trainer 2018-2020, Jes Neal, Community Trainer 2018-2020, T'Sey-Haye Preaster, Project Manager 2018-2020, Vernon Mitchell, Administrative Lead 2017, Timothy Cole, Project Manager 2017, Chris Freeland, Administrative Lead 2016, Desiree Smith-Jones, Project Manager 2016, Dan Chudnov, Data Engineer 2016-2017.