Screenshots suck. But they’re about to get a lot smarter

Screenshots are ubiquitous. From shopping references to memes and from saving cat photos to serious research, they’re a routine and important quality-of-life feature on computers and mobile devices everywhere.

Android 12 should finally get full-page website screenshots, if Google can iron the bugs out before release. Samsung and Apple have had that feature for a while, but their recent patents show an interest in capturing metadata, from live links to site-specific information like street addresses and song titles.

Tap Scroll Capture when you screenshot a Samsung Galaxy, and the phone’s screenshot tool will stitch together a long screenshot to capture an entire website. Browse your screenshots in the Gallery, and you’ll see a Go To Website shortcut, a feature introduced in 2017.

Since iOS 13 in 2019, when you take a screenshot in Safari, a thumbnail pops up allowing you to select Full Page and save the resulting long website capture as a PDF.

Despite the screenshot’s long history (see Timeline below) and the fact that Apple launched mobile screenshots way back in 2007 on the first iPhone, amazingly these features represent the current state of the art when it comes to mobile screen capture. LG, Huawei and others have their own takes on long web page capture, and third-party apps like LongShot can do it for the rest of us.

XDA Developers’ Mishaal Rahman spotted the new scrolling screenshot capture feature - currently very crash-prone – in Android 12 Developer Preview 1, an early iteration of the OS that should launch in autumn 2021.

But while Google plays catch-up, other developers are already looking at making screenshots smarter.

Where next?

Link saving is perhaps the most in-demand smart screenshot feature. Whether you’re a lawyer or academic doing research, or shopping for this season’s coolest lockdown fashions, preserving the links from the page you’re looking at for future reference is vital.

Full, copyable text capture is similarly important, and the usual solutions here are variations on saving to PDF, a long-established technique that’s seeing progressive refinements to more accurately represent the graphical layout of websites.

On desktop computers, specialist screen capture tools such as Fireshot Pro can do both. Mobile is still getting there, with Apple’s long screenshots accurately capturing text, while Samsung devices will link you back to the original page when you view them in the Gallery.

But new and increasingly sophisticated approaches to extracting and attaching relevant metadata to a screenshot are being developed, such as Qualcomm’s unexpectedly interesting ‘Associating a captured screenshot with application-specific metadata that defines a session state of an application contributing image data to the captured screenshot’, which describes a method to restrict capture to the active window and grab relevant metadata, such as a street address, a website URL, the length of a video, a direct link to a chat session, data about a music track being played, and any text deployed in the screenshot.

Apple was recently granted a patent on its ‘Device, method, and graphical user interface for editing screenshot images’, which also describes an approach to storing metadata, such as GPS locations, URLs, device state and image alt text, and provide a way of annotating and storing or sharing both the screenshot and relevant data about it.

The patent suggests that metadata such as URLs and application download links may be shared alongside screenshots. Accompanying diagrams show automatic link creation to accompany a screenshot of a web page sent in an email.

It also details an interface for editing, annotating and sharing screenshots on a touchscreen. Screenshot annotation is already a feature of Markup in iOS, introduced in 2016 with iOS 10, but this more recent patent’s descriptions of metadata handling add a potential new dimension to the future of screenshot sharing on Apple’s portable devices.

Samsung’s abandoned ‘Method and device of tagging links included in a screenshot of webpage’ shows us that the company has been looking into creating screenshots with active links embedded in them, and a system to allow users to select which links to include and how they store that metadata.

With so much prior art surrounding screenshots, many of these patents may prove to be unsustainable, but, like Samsung’s 2013 ‘Electronic device and method for using captured image in electronic device’ patent application, which appears to presage its Go to URL feature and Microsoft’s ‘Capturing website content through capture services’, which similarly seems to describe its Snip & Sketch app, released for Windows in 2019, even these abandoned intellectual property applications can show us the shape of screenshots to come.

Screen Capture: a timeline

• 1981: MS-DOS on the IBM PC 5150 sends text from the screen buffer to your printer when you press the Print Screen key
• 1985: Apple Macintosh System Software 2.0 adds graphical screenshots, with different key combinations to print them or save them to disk
• 1990: Windows 3.0 now saves the entire screen area to your clipboard as a bitmap image when you press print screen, and the current window when you press Alt-Print Screen
• 1990: third-party Window screenshot tool SnagIt lets you save screenshots to disk; it would later introduce screen-area marquee selection capture and long screenshots
• 2000: scrot is released and quickly becomes ubiquitous on UNIX and Linux systems
• 2002: Apple’s Mac OS X Public Beta launches with Grab screenshot tool inherited from NeXTStep
• 2002: the Windows Snipping Tool gets its first release for tablet PCs
• 2006: Windows Vista launches with marquee selection and annotation via the Windows Snipping Tool
• 2007: the iPhone launches with screenshot capabilities
• 2008: Android launches without screenshotting support. People root their phones to add it
• 2011: Valve adds an integrated screenshot tool to Steam, helping to boost the nascent artform of in-game photography
• 2011: Android finally gets integrated screenshotting in version 4 (Ice Cream Sandwich)
• 2012: Windows 8.0 finally allows screenshots to be automatically saved to disk
• 2014-2015: Samsung begins introducing long screenshots to its TouchWiz UI for Android
• 2016: iOS 10 introduces screen annotations in 2016 via its integrated Markup tool
• 2017: Firefox adds in-browser screenshotting, with long screenshot support
• 2017: Google does the same in Chrome, but only as a developer tool
• 2018: Microsoft introduces Snip & Sketch, which adds extra annotation tools and Universal Windows Platform integration
• 2018: macOS 10.14 introduces new Screenshot tool with additional menus and screen recording features
• 2019: iOS 13 adds long screenshot support to Safari

This article was originally published by WIRED UK