Julien Siems | University of Freiburg

About Me

I am a Ph.D. student at the University of Freiburg advised by Prof. Frank Hutter.

After graduating from the University of Freiburg in 2020 with distinction, I was an Applied Science Intern at AWS Berlin working on Neural Network Pruning, I worked on Bayesian Optimization for combinatorial auctions at the University of Zurich, and worked at Prior Labs as AI Scientist.

Research Interests

Linear RNNs / State-Space Models, State-Tracking, In-Context Learning

News

[Sept. 2026] TempoPFN is accepted at NeurIPS 2026.
[Sept. 2026] New preprint “Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention”.
[Feb. 2026] “Learning State-Tracking from Code Using Linear RNNs” accepted at ICLR ICBINB Workshop 2026.
[Jan. 2026] I joined Prior Labs as AI Scientist.
[Oct. 2025] “TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting” accepted at Tabular-AI Workshop at EurIPS 2025.
[Sept. 2025] I am interning at Microsoft Research Cambridge from Sept. to Nov. 2025.
[Sept. 2025] “DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products” is accepted at NeurIPS 2025.
[Mar. 2025] “DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products” is accepted as Oral at FM-WILD Workshop at ICLR 2025.
[Feb. 2025] “Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues” is accepted as an Oral at ICLR 2025.

Seminars / Lectures

[Mar. 2025] State-Tracking in Linear RNNs at the Advances in Sequence modeling from Algorithmic Perspectives Seminar (Youtube) (Slides)
[Jan. 2025] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues at the AutoML Seminar. (Youtube) (Slides)
[Apr. 2024] Is Mamba Capable of In-Context Learning? At the AutoML Online Seminar (Youtube) (Extended Slides)

Publications

arxiv

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Julien Siems*, Riccardo Grazzi*, Korbinian Pöppel*, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein (*equal contribution)

arxiv

PDF Code

ICBINB@ICLR

Learning State-Tracking from Code Using Linear RNNs

Julien Siems*, Riccardo Grazzi*, Kirill Kalinin, Hitesh Ballani, Babak Rahmani* (*equal contribution)

I can't believe it's not better workshop at ICLR 2026.

PDF

NeurIPS

TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting

Vladyslav Moroshan*, Julien Siems*, Arber Zela, Timur Carstensen, Frank Hutter (*equal contribution)

Neural Information Processing Systems (NeurIPS) 2026

PDF Code

NeurIPS

DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products

Julien Siems*, Timur Carstensen*, Arber Zela, Frank Hutter, Massimiliano Pontil, Riccardo Grazzi* (*equal contribution)

Neural Information Processing Systems (NeurIPS) 2025

PDF

ICLR (Oral)

Unlocking State-Tracking in Linear RNNs through Negative Eigenvalues

Riccardo Grazzi*, Julien Siems*, Jörg Franke, Arber Zela, Frank Hutter, Massimiliano Pontil (*equal contribution)

International Conference on Learning Representation 2025 (Oral)

PDF

Interpretability@NeurIPS (Oral)

GAMformer: In-Context Learning for Generalized Additive Models

Andreas Müller*, Julien Siems*, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter (*equal contribution)

NeurIPS Workshop on Interpretable AI Past, Present and Future (Oral)

PDF

TS@NeurIPS (Spotlight)

Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models

Sathya Kamesh Bhethanabhotla*, Omar Swelam*, Julien Siems, David Salinas, Frank Hutter (*equal contribution)

NeurIPS Workshop on Time Series in the Age of Large Models (Spotlight)

PDF

ME-FoMo@ICLR

Is Mamba Capable of In-Context Learning?

Riccardo Grazzi*, Julien Siems*, Simon Schrodi, Thomas Brox, Frank Hutter (*equal contribution)

ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) and AutoML Conference 2024

PDF

NeurIPS

Curve Your Enthusiasm: Concurvity Regularization in Differentiable Generalized Additive Models

Julien Siems*, Konstantin Ditschuneit*, Winfried Ripken*, Alma Lindborg*, Maximilian Schambach, Johannes S. Otterbach, Martin Genzel (*equal contribution)

Neural Information Processing Systems (NeurIPS) 2023

PDF Code

AI4ABM@ICLR

Interpretable Reinforcement Learning via Neural Additive Models for Inventory Management

Julien Siems, Maximilian Schambach, Sebastian Schulze, Johannes S Otterbach

Workshop AI4ABM at International Conference on Learning Representations (ICLR) 2023

PDF

AAAI

Bayesian Optimization-Based Combinatorial Assignment

Jakob Weissteiner*, Jakob Heiss*, Julien Siems*, Sven Seuken (*equal contribution)

AAAI 2023

PDF Code

IJCAI-ECAI

Monotone-Value Neural Networks: Exploiting Preference Monotonicity in Combinatorial Assignment

Jakob Weissteiner*, Jakob Heiss*, Julien Siems*, Sven Seuken (*equal contribution)

International Joint Conference on Artificial Intelligence & European Conference on Artificial Intelligence 2022

PDF Code

ICLR

Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

Arber Zela*, Julien Siems*, Lucas Zimmer*, Jovita Lukasik, Margret Keuper, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2022

PDF Code

AutoML @ ICML

Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise Ratios

Julien Siems, Aaron Klein, Cedric Archambeau, Maren Mahsereci

8th ICML Workshop on Automated Machine Learning (AutoML) at ICML 2021

PDF

ICLR

NAS-Bench-1Shot1: Benchmarking and dissecting one-shot neural architecture search

Arber Zela*, Julien Siems*, Frank Hutter (*equal contribution)

International Conference on Learning Representation 2020

PDF Code

CVIU

Faster training of Mask R-CNN by focusing on instance boundaries

Roland Zimmermann*, Julien Siems* (*equal contribution)

Computer Vision and Image Understanding Journal

PDF Code