Tomas Hrycej
Title
Dr.
Last Name
Hrycej
First name
Tomas
Email
tomas.hrycej@unisg.ch
21 results
Now showing 1 - 10 of 21
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder(IEEE, 2026-05-06); ; ; Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30 % of the training data is sufficient to reach approximately 90 % of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.Type:conference paperJournal:2026 IEEE Swiss Conference on Data Science and AI (SDS) - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Subset Pretraining for Enhancing Neural Network Training EfficiencyWe propose a novel alternative to traditional randomly sampled mini-batches for gradient computation: using a fixed subset for complete pretraining of a neural network model. This approach enables deterministic convergence instead of a merely probabilistic one, as proven by the stochastic approximation theory, whose prerequisites are frequently violated by popular optimization algorithms. The approach is justified by the hypothesis that the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing with the whole training set. They are also compatible with efficient second-order optimization methods, such as the conjugate gradient optimizer. These methods are particularly efficient in the convex environment of the loss minimum. The image classification datasets MNIST, CIFAR-10, and CIFAR-100, (optionally extended by augmentation of training data) test this hypothesis. The experiments confirm that the models achieve performance equivalent to that when trained with the conventional training scheme. In conclusion, if the overdetermination ratio for the given model and dataset sufficiently exceed unity, even small subsets are representative. This results in a possible reduction of the computing expense to a tenth or less. This paper is an extended version of Spörer et al. [13].Type:conference paperJournal:Communications in Computer and Information ScienceVolume:2703 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Is There an Optimal Depth of Residual Networks?(Springer Nature (Switzerland), 2025-05-03); ; Although deep neural networks have given the name to the domain of "deep learning", it is still an unresolved question of which depth is the best choice for a given task. A large depth brings problems with the convergence of gradientbased learning algorithms through the phenomenon of vanishing gradient. On the other hand, it is a widespread opinion that a small depth has insufficient representational power for many tasks. The discovery of the concept of residual connections-an identity mapping parallel to a conventional layer-has alleviated the convergence problem so that the discussion of optimum depth lost a part of its motivation, resulting in the assumption of "the more the better". The work presented here shows that a shallow architecture of parallel layers has comparable expressive power as a deep stack of residual layers. This is theoretically justified by expanding the residual layer stack analogical to the Taylor expansion, truncating the higher-order terms into a single broad layer composed of original layers in parallel. This hypothesis has been confirmed by computing experiments with the widespread computer vision benchmark datasets MNIST and CIFAR-10. The 6,912 runs have shown that the shallow and the deep architectures do not substantially differ in performance on both training and validation sets if the total number of parameters is equal. The rough equivalence of the two extreme (deep and shallow) architectures suggests the possibility that an intermediary architecture may be superior. Another series of computing experiments disclosed that the performance does not substantially differ even then. The conclusion is that the performance of an architecture depends more substantially on the total number of parameters than on the sequential or parallel connection of layers.Type:conference paperJournal:Communications in Computer and Information ScienceVolume:2454 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, A Convexity-dependent Two-Phase Training Algorithm for Deep Neural Networks(SciTePress, 2025-10-27); ; ; ; The key task of machine learning is to minimize the loss function that measures the model fit to the training data. The numerical methods to do this efficiently depend on the properties of the loss function. The most decisive among these properties is the convexity or non-convexity of the loss function. The fact that the loss function can have, and frequently has, non-convex regions has led to a widespread commitment to non-convex methods such as Adam. However, a local minimum implies that, in some environment around it, the function is convex. In this environment, second-order minimizing methods such as the Conjugate Gradient (CG) give a guaranteed superlinear convergence. We propose a novel framework grounded in the hypothesis that loss functions in real-world tasks swap from initial non-convexity to convexity towards the optimum - a property we leverage to design an innovative two-phase optimization algorithm. The presented algorithm detects the swap point by observing the gradien t norm dependence on the loss. In these regions, non-convex (Adam) and convex (CG) algorithms are used, respectively. Computing experiments confirm the hypothesis that this simple convexity structure is frequent enough to be practically exploited to substantially improve convergence and accuracy.Type:conference paperJournal:Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge ManagementVolume:1 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Integrating the Attention Mechanism Into State Space Models(2025-07-18); ; Type:conference paperJournal:2025 IEEE Swiss Conference on Data Science (SDS) - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Reducing the Transformer Architecture to a Minimum(SCITEPRESS - Science and Technology Publications, 2024-11); ; ; Scopus© Citations 2 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Reducing the Transformer Architecture to a Minimum(SciTePress, 2024-11); ; ; Scopus© Citations 2 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Efficient Neural Network Training via Subset Pretraining(SCITEPRESS - Science and Technology Publications, 2024-11-22); ; ; Scopus© Citations 3 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Efficient Neural Network Training via Subset Pretraining(SciTePress, 2024-11); ; Scopus© Citations 3 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Make Deep Networks Shallow AgainDeep neural networks have a good success record and are thus viewed as the best architecture choice for complex applications. Their main shortcoming has been, for a long time, the vanishing gradient which prevented the numerical optimization algorithms from acceptable convergence. An important special case of network architecture, frequently used in computer vision applications, consists of using a stack of layers of the same dimension. For this architecture, a breakthrough has been achieved by the concept of residual connections-an identity mapping parallel to a conventional layer. This concept substantially alleviates the vanishing gradient problem and is thus widely used. The focus of this paper is to show the possibility of substituting the deep stack of residual layers with a shallow architecture with comparable expressive power and similarly good convergence properties. A stack of residual layers can be expressed as an expansion of terms similar to the Taylor expansion. This expansion suggests the possibility of truncating the higher-order terms and receiving an architecture consisting of a single broad layer composed of all initially stacked layers in parallel. In other words, a sequential deep architecture is substituted by a parallel shallow one. Prompted by this theory, we investigated the performance capabilities of the parallel architecture in comparison to the sequential one. The computer vision datasets MNIST and CIFAR10 were used to train both architectures for a total of 6,912 combinations of varying numbers of convolutional layers, numbers of filters, kernel sizes, and other meta parameters. Our findings demonstrate a surprising equivalence between the deep (sequential) and shallow (parallel) architectures. Both layouts produced similar results in terms of training and validation set loss. This discovery implies that a wide, shallow architecture can potentially replace a deep network without sacrificing performance. Such substitution has the potential to simplify network architectures, improve optimization efficiency, and accelerate the training process.Type:conference paperJournal:Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge ManagementScopus© Citations 1
- «
- 1 (current)
- 2
- 3
- »