Multi-Scale Context Aggregation by Dilated Convolutions

This research paper explores improvements to semantic segmentation, a computer vision task involving pixel-wise image classification. The authors introduce a novel convolutional network module utilising dilated convolutions to efficiently aggregate multi-scale contextual information without resolution loss. They also present a simplified, higher-performing front-end prediction module derived from an image classification network. Experiments on various datasets demonstrate the efficacy of the proposed module in boosting the accuracy of state-of-the-art semantic segmentation systems. The enhanced performance stems from both the novel module and the simplification of pre-existing network components.

Deep Residual Learning for Image Recognition

This research paper introduces a deep residual learning framework for image recognition, addressing the degradation problem encountered when training extremely deep neural networks. The authors propose reformulating network layers to learn residual functions, improving optimisation and achieving significant accuracy gains. Their approach, involving "shortcut connections," won first place in several ILSVRC & COCO 2015 competitions. Extensive experiments on ImageNet and CIFAR-10 datasets demonstrate the effectiveness of this method, even with networks exceeding 1000 layers. The superior performance is attributed to easier optimisation and the learning of more effective representations.

Order Matters: Sequence to Sequence for Sets

This research paper explores the impact of input and output ordering on sequence-to-sequence (seq2seq) models, particularly when dealing with sets rather than sequences. The authors demonstrate that data order significantly affects model performance, even with powerful models like LSTMs. They propose a "Read-Process-Write" architecture to handle unordered input sets and a training algorithm that searches for optimal output orderings during training. Experiments on tasks like sorting and language modelling support their findings, highlighting the importance of considering data order for improved seq2seq model performance. The paper concludes that careful consideration of input and output ordering is crucial for maximising the performance of seq2seq models, especially when dealing with sets.

ImageNet Classification with Deep Convolutional Neural Networks

This research paper details the creation and training of a deep convolutional neural network (CNN) for large-scale image classification. The authors achieved state-of-the-art results on the ImageNet dataset by employing several innovative techniques, including ReLU nonlinearities, training across multiple GPUs, local response normalisation, and dropout regularisation. The architecture of this substantial network, with its multiple convolutional and fully-connected layers, is described in detail, along with the data augmentation methods used to mitigate overfitting. The paper concludes by presenting the impressive classification accuracy achieved and discusses potential future improvements.

Recurrent Neural Network Regularization

This research paper explores a novel application of dropout regularization to Long Short-Term Memory (LSTM) networks, a type of Recurrent Neural Network (RNN). The authors demonstrate that applying dropout to specific connections within LSTMs significantly reduces overfitting, improving performance on various tasks including language modelling, speech recognition, machine translation, and image caption generation. Their method addresses the limitations of previous dropout applications to RNNs which amplified noise and hindered learning. Empirical results across multiple datasets showcase substantial performance gains compared to non-regularized LSTMs and some existing state-of-the-art models. The paper concludes that this improved dropout implementation makes it a valuable tool for enhancing the performance of RNNs in diverse applications.

The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy

This blog post explores the capabilities of Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) networks. The author demonstrates RNNs' ability to generate text character by character after training on various datasets, including Paul Graham's essays, Shakespeare's works, Wikipedia articles, LaTeX code, and Linux source code. The examples showcase the RNN's capacity to learn complex syntactic structures and patterns from raw data. Furthermore, the author investigates the internal workings of RNNs through visualisations of neuron firings and prediction distributions, highlighting the emergent properties of these models. Finally, the post discusses current research directions in RNNs, focusing on attention mechanisms and their application to various fields.

Minimizing Description Length in Neural Networks

This paper proposes a novel method for training neural networks that minimises the description length of the weights, improving generalisation, particularly with limited training data. The approach uses noisy weights and models their distribution with a mixture of Gaussians, adapting both weight precision and probability density during training. This method is compared to standard weight decay, showing potential advantages in handling high-dimensional data with scarce training examples. The core idea is to apply the Minimum Description Length principle, balancing the complexity of the model with the accuracy of the fit to the training data. Exact derivatives are computed efficiently, avoiding time-consuming Monte Carlo simulations, for networks with one hidden layer and linear output units.

Pointer Networks

This paper introduces Pointer Networks (Ptr-Nets), a novel neural architecture designed to predict output sequences whose elements correspond to positions within an input sequence. Unlike previous sequence-to-sequence models, Ptr-Nets handle variable-length output dictionaries by employing an attention mechanism as a pointer to select input elements. The effectiveness of Ptr-Nets is demonstrated through their application to three geometric problems: finding convex hulls, computing Delaunay triangulations, and solving the travelling salesman problem. The results show Ptr-Nets outperform existing methods and generalise to input sizes beyond those seen during training. The authors conclude by suggesting further applications for this architecture in other combinatorial optimisation problems.

Minimum Description Length (MDL) Principle

This following provides a mathematically precise introduction to the Minimum Description Length (MDL) principle, a method for inductive inference and model selection. It begins with foundational concepts in information theory and statistics, then explains a "crude" two-part code version of MDL before moving to a more sophisticated "refined" version based on universal coding. The refined approach is explored through various interpretations, including compression, counting, Bayesian, and prequential perspectives. Finally, the tutorial discusses MDL's relationship to other inferential approaches and addresses potential limitations.

Building Effective LLMs Agents

This Anthropic blog post details best practices for building effective large language model (LLM) agents. It distinguishes between simpler workflows and more complex, autonomous agents, outlining various workflow patterns like prompt chaining and parallelization. The post advocates for starting with simple solutions and increasing complexity only when necessary, emphasising the importance of clear tool documentation and thorough testing for reliable agent performance. It also provides examples of successful agent applications in customer support and software coding, highlighting the value of agent-computer interface (ACI) design. Finally, the piece stresses simplicity, transparency, and well-documented tools as key principles for building effective and trustworthy agents.

2025 Tech Predictions

The text presents a compilation of predictions for significant technological and economic trends in 2025, gathered from fifty leading technology experts. The majority focus on various facets of artificial intelligence, including the rise of AI agents, new interfaces, and widespread applications across numerous sectors. Other predictions explore developments in consumer products, healthcare, finance (particularly crypto-fintech), and the broader economic landscape, including the roles of data centres and government regulation. Several contributors also forecast shifts in business models and software development practices.

Quantifying Complexity in Closed Systems: The Coffee Automaton

This paper investigates the quantification of complexity in closed systems, using a simulated coffee-cream mixing model as a case study. The authors explore different complexity measures—apparent complexity, sophistication, logical depth, and light-cone complexity—comparing their strengths and weaknesses. They conduct numerical experiments using a cellular automaton, finding that complexity initially rises and then falls as entropy increases, mirroring the universe's evolution. The study focuses on approximating Kolmogorov complexity through coarse-graining techniques to overcome computational limitations, ultimately suggesting apparent complexity provides the most intuitive measure in this context. Further research directions are proposed to improve the accuracy and theoretical understanding of complexity in such systems.

GPipe: Scaling Deep Neural Networks

The paper introduces GPipe, a novel library for efficiently training extremely large neural networks. GPipe achieves this through a pipeline parallelism approach that splits a training batch into micro-batches, processing them concurrently across multiple accelerators. This method, combined with re-materialisation to reduce memory demands, allows for near-linear scaling of training speed with the number of accelerators. The paper demonstrates GPipe's effectiveness on both image classification (using AmoebaNet) and machine translation (using a massive multilingual Transformer model), showcasing significant performance improvements compared to existing methods. The authors also analyse GPipe's performance characteristics and compare it to alternative model parallelism techniques.

Message Passing Neural Networks for Quantum Chemistry

This research paper explores Message Passing Neural Networks (MPNNs) for predicting the quantum mechanical properties of molecules. The authors introduce a unified framework encompassing various existing MPNN models, improving upon them to achieve state-of-the-art results on the QM9 dataset. They investigate different MPNN variations, focusing on message functions, readout functions, and input representations, and introduce a novel "towers" approach to improve computational efficiency and generalisation. The study highlights the potential of MPNNs for accurate and efficient chemical property prediction, surpassing traditional methods that rely on hand-engineered features.

Machine Super Intelligence by Shane Legg

This thesis explores universal artificial intelligence (UAI), focusing on Hutter's AIXI agent as a theoretical model of highly intelligent systems. It examines existing definitions and measurements of intelligence, proposing a novel universal intelligence measure (Υ) based on AIXI's performance across a broad range of computable environments. The work also investigates the computational limits of UAI and introduces a new temporal difference learning algorithm without a learning rate. Finally, it discusses the feasibility and potential implications of creating superintelligent machines, considering both theoretical and practical approaches.

The First Law of Complexodynamics

This text proposes a "First Law of Complexodynamics" to explain why the complexity of physical systems increases, then decreases, unlike entropy which only increases. It suggests using a modified concept of Kolmogorov complexity, termed "complextropy," to measure this complexity. The author argues that complextropy, unlike standard measures, considers computational resource constraints to accurately capture the complexity of systems at various stages of evolution, such as a mixing coffee cup. This is proposed as a solution to a question posed by Sean Carroll regarding the relationship between entropy and complexity. The author outlines open problems and potential research directions to formally define and prove this law.

DeepSeek-V3: A 671B Parameter Mixture-of-Experts Language Model

The document details DeepSeek-V3, a 671-billion parameter Mixture-of-Experts large language model. It covers its architecture, including a novel auxiliary-loss-free load balancing strategy and a multi-token prediction objective. The text describes the model's training process, infrastructure (using 2048 NVIDIA H800 GPUs and an FP8 mixed-precision framework), and post-training methods like supervised fine-tuning and reinforcement learning. Extensive evaluations demonstrate DeepSeek-V3's strong performance across various benchmarks, exceeding many open-source models and rivaling closed-source models in many areas, while maintaining remarkably low training costs. Finally, the paper discusses limitations and future research directions.

DeepSeekMath: Enhanced Mathematical Reasoning in LLMs

This research paper introduces DeepSeekMath, a large language model specifically designed for mathematical reasoning. The model is pre-trained on a massive, high-quality dataset of mathematical text extracted from Common Crawl, exceeding the size of comparable datasets. A novel reinforcement learning algorithm, Group Relative Policy Optimization (GRPO), further enhances DeepSeekMath's performance. Extensive benchmarking demonstrates DeepSeekMath's superior capabilities compared to other open-source models, achieving accuracy close to that of leading closed-source models on various mathematical reasoning tasks. The paper also analyses the effects of code pre-training and the use of arXiv data in improving mathematical reasoning capabilities of LLMs.