Scaling Scaling: Stochastic Variational Inference for X-ray Diffraction Data

Inaugural MLCV Seminar

The systematic errors inherent in crystallographic data introduce a core challenge to structure determination. These errors result from numerous processes including absorption by the sample and surrounding material, polarization, and imperfections in the crystal lattice. Prior to merging redundant observations, these artifacts are typically corrected by an optimization algorithm known as scaling. Recently, we showed that the two procedures, scaling and merging, could be unified into a single algorithm based on Bayesian (variational) inference and deep learning (Dalton et al., 2022). While this algorithm compared favorably to other algorithms for a broad range of experimental designs, it was limited in applicability by its memory requirements. This was particularly prohibitive for serial crystallography wherein the number of diffraction images becomes too large to fit in memory. In this talk, I will introduce a solution to this problem. By adding a pooling mechanism to the neural scaling model from our previous work, it is possible to train the model on small batches of data. Thereby the memory requirements are determined by the batch size rather than the size of the full data set. This innovation opens the door to treating very large datasets with hundreds of thousands or millions of diffraction images.

Recording

References

2022

Crystallography

A unifying Bayesian framework for merging X-ray diffraction data

Kevin M. Dalton, Jack B. Greisman, and Doeke R. Hekstra

2022

Abs DOI

Novel X-ray methods are transforming the study of the functional dynamics of biomolecules. Key to this revolution is detection of often subtle conformational changes from diffraction data. Diffraction data contain patterns of bright spots known as reflections. To compute the electron density of a molecule, the intensity of each reflection must be estimated, and redundant observations reduced to consensus intensities. Systematic effects, however, lead to the measurement of equivalent reflections on different scales, corrupting observation of changes in electron density. Here, we present a modern Bayesian solution to this problem, which uses deep learning and variational inference to simultaneously rescale and merge reflection observations. We successfully apply this method to monochromatic and polychromatic single-crystal diffraction data, as well as serial femtosecond crystallography data. We find that this approach is applicable to the analysis of many types of diffraction experiments, while accurately and sensitively detecting subtle dynamics and anomalous scattering.