(Last updated Dec 2025)

In April 2025, PIRC members began a literature scan related to interactive, executable research content, broadly querying for other relevant reviews, survey, projects, and efforts, including articles and resources that discuss preservation of interactive environments but also those that could serve as a model for group’s development in terms of our eventual framework. This scan was envisioned as a living document, and was last updated in December 2025. Some sections are underdeveloped as noted below and might be the focus of future work.

Like any literature scan, we wanted to understand what past work has or has not covered, and also confirm that we were headed in a useful direction. One challenge that was apparent immediately is that there is not one intellectual home or foundation for this topic. It is also not the type of topic that relies solely on peer reviewed scholarly articles––a mix of proposals, preprints, conference proceedings, presentations, websites, repositories all pepper the landscape.

The resources identified generally fit into these categories:

• Initiatives to promote interactivity

• Open science concepts, frameworks, and best practices

• Technical solutions

• Preservation

• Broader conceptual models

Initiatives to promote interactivity¶

The proposal that scientific communication should move beyond static outputs is not new. There have been many efforts pre-dating ours which attest to frustration with static content as well as the need to link research outputs (methods, data, code, narrative). Some prior work describes large-scale organizational efforts to propose new technology and models while other work surveys user groups to benchmark what tools and software researchers actually use. However smaller scale efforts are as common, focusing on a particular domain or even a personal proposal to offer a solution. For PIRC’s purposes, a deep focus on past efforts is not warranted, but three meetings stand out as important events which moved the conversation on interactive publications forward: Executable Paper Grand Challenge/2011 International Conference on Computational Science; Beyond the PDF; and AGU Notebooks Now.

Elsevier’s Executable Paper Grand Challenge in 2011 which sought proposals for an Executable Paper concept with the primary aim of better representing data intensive work within a journal article. Wilson, 2012 includes a full list of finalists to the Challenge. The prizewinner, the Collage Authoring Environment Nowakowski, 2011 emerged from this Challenge as an early prototype for embedding executable code within a narrative that can later be rendered and manipulated. Cushing builds on the work of the Grand Challenge and discusses the challenges involved in operationalizing executable papers while also noting the value of an executable paper for scientific reproducibility. Cushing also discusses the role of the main actors involved in the lifecycle of an executable paper–– author, publisher, reviewer, infrastructure provider, reader. Hinsen, 2011 also proposes a design for an executable format based on a model of data, code and text that documents provenance, and verifies and authenticates results. These early proposals offered interesting directions but most did not come to fruition; and also focused more on the execution of the paper in real time and not on preservation for later access.

Beyond the PDF held in 2011 and as BTPDF2 in 2013 were precursor meetings to Force11 to which this working group reports. The goal of these meetings, literally to move beyond PDFs, incorporated conversations about publishing workflows and possibilities for digital scholarship. Presentations included proposals for tools that allowed for collaboration and linking data with text (ex. Sefton, 2010). These meetings spun off several initiatives related to opening scholarship, data citation, and literature discovery and citation management. (Beyond the PDF2: Outcomes – FORCE11, 2013)

More recently, AGU’s Notebooks Now initiative focused on creating a full workflow to support computational notebooks as primary publications Stall, 2022. Initial work at a 2022 stakeholder meeting led to a pilot focused operationalizing the process of submitting and hosting interactive content. (Research Articles - AGU Notebooks Now, n.d.) Perez’s evaluation of the initiative (Pérez, 2025) found it to be a foundation for open workflows without any preference to particular vendors or adherence to proprietary systems. The Notebooks Now pilot was completed by Curvenote (https://curvenote.com) and is hosted at https://notebooks.agu.org, while a successful demonstration of the possibilities of publishing notebooks, did not address downstream preservation or archival concerns.

Kramer & Bosman, 2016’s survey on research tool usage inventories tools used across the research workflow. Although it is now 10 years old, the findings are essential for understanding the tools and applications used by researchers that would form the foundation for an interactive publication. While survey respondents reported the components of their workflows, not all applications would be required to recreate published results. More recently (Kanza et al., 2023) executed a smaller survey asking how respondents organize and link work to assess requirements for a digital research environment. A mix of tools and products were reported with no one solution.

Around the same time, other efforts emerged to package experiments and outputs explicitly for evaluation and reuse by other researchers such as ReproZip (Chirigati et al., 2016), Research Objects (Bechhofer et al., 2013) and its more current iteration RO-Crate (Soiland-Reyes et al., 2022). RO-Crate focuses on packaging research artefacts and their essential metadata together. Whole Tale (Brinckman et al., 2019) adopted a similar approach with the goal of building a platform to host executable research objects along with their data, code and environments to support reproducibility.

Several other works discuss attempts to create executable papers, from computational notebook based articles (Chandre & Dubois, 2021), PDFs with interactive map layers (Štular & Lozić, 2023), personal accounts of creating an executable paper (Lasser, 2020), and through the idea of a binding, somewhat inline with early discussions of research objects but which connects research component for inspecting, manipulation and review (Konkol et al., 2019).

The most successful of these early efforts have brought people together to brainstorm new functionality and new ways of communicating, but none of them to date have led to long term transformation or have addressed persistence of these new formats.

Open science concepts, frameworks, and best practices¶

Open science has emerged in recent years as a goal and an expectation of research outputs. Interactive research content benefits from open science practices by promising transparent access to research, enabling readers to look under the hood, manipulate inputs and question outputs. We likely do not have a comprehensive bibliography, but we have gathered enough references to follow a thread from efforts to open data to later extensions with software and code (Morin et al., 2012) as well as workflows overall Atkinson et al., 2017; Gil et al., 2007. There are many attempts to ground individual projects in ideas of transparency, reproducibility, and the FAIR principles. There are parallel conversations with research sharing and curation in terms of licensing, venues for deposit, funder, publisher and institutional support (Baillieul et al., 2018). Data sharing comes across as a rising success story but also a necessary component to be shared if executable content will be effective long term. The extensions of data sharing to code sharing seem to claim that (at least around 2020-25), data sharing is successful or can be, but code falls short (Jean-Quartier et al., 2025; Janssen, 2017), and our infrastructure is not adequate to support and preserve code and other ephemera hosted on sites like Github (Escamilla et al., 2022).

Reproducibility¶

Under the open science umbrella, reproducibility weaves many of these disparate threads together. Enabling interactivity may be one approach to ensuring reproducibility, or at least replicability by quickly allowing new inputs to be tested. There are also various working groups, emerging standards, infrastructure and best practices that address reproducibility. The reproducibility crisis is well covered, and not the focus of this group, but two streams of best practices for notebooks could be helpful - one, related to technical documentation to make notebooks re-executable with considerations for dependencies, libraries, etc (Samuel & Mietchen, 2024; Nüst et al., 2020; Ziemann et al., 2023; CEOS Working group on Information Systems and Services, 2025 and others), and two, better ways to reproduce a legible narrative. Many papers highlight reproducibility through executable research artifacts, reflecting growing expectations for transparent methods and verifiable results, but they don’t necessarily address integration of these outputs.

Behavioral norms¶

Some works focus on documenting behavior, norms, and incentives. There are several interesting approaches to identifying relevant audiences or personas and identifying challenges that prevent them from adopting new tools. Gomes et al., 2022 lists challenges that get in the way of people sharing data and code which range from more technical stops to psychological reasons. These challenges stood out: difficulty with complex workflows and large data; dealing with sensitive content; feeling insecure about sharing one’s code. VandenBosch et al., 2023 mentions training researchers toward good end user deposit behavior as a necessary prerequisite for accessioning notebooks into repositories.

Borgman & Groth (2025) address the idea of reuse (for data) though this is relevant to PIRC in terms of who might be accessing and reusing executable content; in particular, the theory of “distance” between creators and reusers and “questions of how, why, for whom and for how long data are reusable” apply to interactive content. The authors’ point that “reusing datasets for purposes similar to those for which they were created is usually much easier than redeploying them for new purposes” also extends to graceful degradation.

Technical Solutions¶

Containers, Notebooks and Notebooks servers¶

Many efforts exist related to Docker, Binder, Containers, and general research environment solutions to keep all research products together for an end user to resurface. Mecum et al., 2018 proposes DataOne Data Package as a standard to group together the digital artifacts of scientific research into a packaging format that preserves provenance and makes use of other open standards. These are useful and can potentially be read as “solving the problem” of serving interactive content but as some articles mention, there are still adoption and training issues (Ziemann et al., 2023; Boettiger, 2015). Several resources address the challenge of archiving dynamic, computational outputs (e.g., notebooks, software environments, interactive visualizations). Concerns center on emulation, documentation, and sustainable infrastructure (Ferlanti et al., 2023). Singer, 2020 discusses the popularity of Jupyter notebooks while also noting both incidental and intrinsic challenges, the most relevant to PIRC being: no versioning due to the structure of notebooks and no introspection in that the notebook code has no direct links between documentation/markdown cells and source code. There has been a lot of movement on the technical aspects of reproducibility. What key works have we missed?

Machine-readable metadata standards¶

Different domains and subfields rely on or reference standards and bodies that have already developed frameworks; we should be careful not to reinvent the wheel when there are many established standards that we might apply. Leipzig et al., 2021 offers a review of metadata standards relevant to reproducible computational research across an “analytic stack” consisting of input data, tools, reports, pipelines, and publications. Other potentially relevant standards to consider should we move beyond a conceptual model: FAIR, TRUST, OAIS, CodeMeta.

Domain specific solutions¶

There is much variability across domains, but many works offer specific computational solutions according to disciplinary needs (Electronic Lab Notebooks, protocols, methods heavy fields). ELN evolution could be an interesting model despite being a pretty different use case for experiment design and initial data collection (Higgins et al., 2022). Biocontainers offer an example of a framework for maintaining multiple versions of bioinformatics software for complex analyses (Da Veiga Leprevost et al., 2017). Literature on multiverse analyses also provides an interesting use case for interactive research content that enables alternate paths of exploration within an analysis (Dragicevic et al., 2019, Sarma et al., 2023, Steegen et al., 2016, Liu et al., 2021). Simson, 2024 specifically tackles the challenges of turning a static work into an interactive paper, including data file size and lack of interactive table functionality within his chosen software. There are likely many more scenarios available, which could be considered as we pursue case studies at a later date.

Preservation¶

Preservation of software¶

Our list includes many resources related to repositories and how they do or do not serve interactive formats. VandenBosch et al., 2023 asks: can repositories handle notebooks and their associated data and dependencies? Efforts by CERN/Zenodo and the Force11 software citation and attribution group, deposit guidelines for software digital assets by Software Sustainability Institute, and the work of JOSS are well-described in this article. There is also a substantive body of work related to software preservation within repositories, what should be preserved and how (Barker et al., 2022; Matthews et al., 2010). Much of this work feels a bit out of scope or even settled. One relevant excerpt comes from Rios, 2016: There are variations on what preservation scales are expected. Some repositories promise 10-20 years, but with software in particular, some expectations from researchers are as little as 5 years. Ultimately what we were most interested in are examples of long term access and usability of interactive content but maybe the long term hasn’t occurred yet.

Herterich, 2019 discusses levels of preservation (convert to stable format/PDF; limited form of access; or fully actionable notebook with complete documentation) that we might consult. Chassanoff & Altman, 2020 offer a “curation model and decision framework for preserving research software as a scholarly object” (Table 1) with six areas of focus for curatorial preservation activities. They suggest future directions of research “including methods for extracting legacy content, development of digital preservation metadata and associated workflows, and understanding what different communities of practice require when reusing born-digital content,” all of which are relevant to interactive content.

Preservation of websites and Digital humanities and preservation of digital media¶

There are a range of research and academic library efforts to preserve digital content that might be applicable (Day et al., 2018, Greenberg et al., 2025), but we have not delved too deeply into this area. The preservation of digital humanities projects could also be good models for dynamic content (VandeCreek, 2024) to be explored at a later date.

Conceptual models and future considerations¶

Many of these articles offer conceptual models that PIRC has considered incorporating into the framework for the degradation of interactive content. Each on its own are strong but together they helped us form a more cohesive model of our own. Carpenter, 2022 provides background for early visions of linked digital content, many of which still apply: Multi-format and multimedia when appropriate; Interoperable with other resources; Machine-readable; Adaptive in design for different displays; Accessible for people with different capabilities; Transformable to other forms or environments as needed; Atomize-able, so users can reuse components; Described with high-quality metadata; Preservable; Persistently Linkable; Trackable. Figure 2, “Format Transitions in the Future,” from 1992-2032 details some categories of research output that lend themselves to increasing interactivity (with more static precursors) that are helpful building blocks for the GD framework.

Tanimoto, 1990 presents the concept of “levels of liveliness” in visual programming systems. Figure 2 offers four levels from (1) a description/flowchart as the most static level, (2) executable specification, (3) responsive to user edits, and (4) continual updates or streams. The first three mirror PIRC’s GD proposal closely.

Arenas et al., 2019 focuses on cloud environments for data science at scale. The discussion of target users is helpful (researcher, investigator, referee, dataset provider, program manager, project manager, system manager) as we look forward to sharing the GD framework. Figure 3 offers recommendations for four tiers of controlled access to research areas and products. Controlled user access is not an inherent goal of GD, but the color coded tiers provide some useful information.

Treloar & Klump, 2019’s Figure 4 details five layers within “Object Curation Domains” the object, storage, context, provenance and archival layers moving from private to collaborative to publication domains. The layers within the Publication Domain are useful to consider as we think about how research outputs can move from progressive interactivity to graceful degradation. Are there parallels between the object layer to level 3, context and provenance layers to level 2, and the archival layer to level 1?

We approached this literature scan broadly, and consider it a living document whose underlying resources can be shared and updated. The framework for the preservation of interactive research content rests on much work that precedes it though we feel that we have carved out a new model that encourages exploration and experimentation.