Donald Sebastian Leung

Infrastructure | Ops

Send me an email

  • Migrating PyTorch deep learning workloads to the Ascend ecosystem

    +

    +

    +

    +

    +

    +

    The notebook source code for this article is available on GitHub.

    PyTorch is an open source deep learning framework in Python originally developed by Meta, now hosted under the vendor-neutral PyTorch Foundation. It is the de-facto industry standard for training modern deep learning models and large language models (LLMs). Like most other deep learning frameworks, PyTorch natively supports accelerating machine learning workloads on NVIDIA GPUs through its CUDA ecosystem without the need for special plugins or adapters.

    (more…)

    +

    +

    +

    +

    +

    +

    + , , , , , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Accelerating model inference on NVIDIA Tesla T4 with ONNX Runtime

    +

    +

    +

    +

    +

    +

    This Kaggle Notebook is a GPU-enabled variation of my original notebook experiment.

    Open Neural Network Exchange (ONNX) is an open format built to represent machine learning models. It is used by popular deep learning frameworks such as MindSpore, PyTorch and TensorFlow to export machine learning models trained by these frameworks into the portable, unified and vendor-neutral ONNX format. This allows exported models to be loaded by any compatible framework or runtime for inference instead of being locked in to a single framework or provider.

    In this notebook experiment, we will load an ONNX model trained on the CIFAR-10 dataset mounted at /kaggle/input/models/donaldsebleung/cifar10-linear-simple/onnx/default/1/cifar10-linear-simple.onnx and serve it with ONNX Runtime for inference. ONNX Runtime is a cross-platform inference accelerator for ONNX models developed by Microsoft under the permissive MIT license.

    (more…)

    +

    +

    +

    +

    +

    +

    + , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Serving machine learning models for inference with ONNX Runtime

    +

    +

    +

    +

    +

    +

    The notebook source code for this experiment is available on GitHub.

    Open Neural Network Exchange (ONNX) is an open format built to represent machine learning models. It is used by popular deep learning frameworks such as MindSpore, PyTorch and TensorFlow to export machine learning models trained by these frameworks into the portable, unified and vendor-neutral ONNX format. This allows exported models to be loaded by any compatible framework or runtime for inference instead of being locked in to a single framework or provider.

    In this notebook experiment, we will download an ONNX model trained on the CIFAR-10 dataset from Hugging Face and serve it with ONNX Runtime for inference. ONNX Runtime is a cross-platform inference accelerator for ONNX models developed by Microsoft under the permissive MIT license.

    (more…)

    +

    +

    +

    +

    +

    +

    + , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Predicting California house prices with MindSpore

    +

    +

    +

    +

    +

    +

    The notebook source code for this article is available on: GitCode, GitHub

    MindSpore is an open source deep learning framework lead by Huawei and the MindSpore community. It is optimized for performing model training and inference tasks on Huawei’s Ascend series processors (NPUs), though it supports NVIDIA GPUs and CPU-only environments as well. It provides a compelling alternative to PyTorch, the leading open source deep learning framework under the Linux Foundation. Furthermore, it provides utility classes and functions compatible with a subset of PyTorch to ease the migration of training and inference pipelines from PyTorch to MindSpore, which is out of scope for this article.

    In this lab, we will use MindSpore for training a simple linear regression model consisting of a single fully connected layer to predict house prices in California. The dataset we will be using is the California housing dataset which we’ll fetch using scikit-learn’s sklearn.datasets.fetch_california_housing function.

    (more…)

    +

    +

    +

    +

    +

    +

    + , , , , , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Exploring the Ascend ecosystem with OrangePi AIpro (20T)

    +

    +

    +

    +

    +

    +

    OrangePi AIpro (20T) is a development board under the Orange Pi brand owned by Xunlong Software and released in 2024. It features 4 CPU cores plus a neural processing unit (NPU) based on Huawei’s Ascend technology capable of running modern AI/ML workloads. The embedded NPU is capable of performing 20 TOPS or trillion operations per second.

    I purchased the OrangePi AIpro (20T) in January 2026 to explore the emerging field of LLM deployment, inferencing and fine-tuning as an infrastructure and platform engineer, as well as exploring the Ascend CANN ecosystem which is swiftly thriving and positioning itself as a strategic alternative to NVIDIA’s CUDA ecosystem in 2026.

    Follow me as I unpack and set up my OrangePi AIpro (20T) development board for exploring AI/ML applications and use cases.

    (more…)

    +

    +

    +

    +

    +

    +

    + , , , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Deploying DeepSeek-R1 for distributed inferencing with Ray

    +

    +

    +

    +

    +

    +

    Ray is an open source framework for scaling Python applications through distributed computing. The framework focuses on scaling AI/ML training and inference pipelines horizontally through distributed resource allocation and execution, though it can be used in other Python projects as well.

    Since October 2025, Ray joined the PyTorch Foundation alongside leading open source AI/ML projects such as PyTorch and vLLM, ensuring its sustained development in the AI/ML ecosystem and vendor-neutral governance.

    Follow me as I deploy a distilled variant of DeepSeek-R1 across 2 GPU-enabled cloud servers on Huawei Cloud with vLLM and Ray. Leveraging Ray as the framework for distributed inference enables us to scale horizontally when needed and increase the number of concurrent requests our model can handle compared to a single server.

    (more…)

    +

    +

    +

    +

    +

    +

    + , ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Running DeepSeek-R1 on a commodity laptop with vLLM

    +

    +

    +

    +

    +

    +

    vLLM is a model serving runtime supporting the most popular families of large language models (LLMs) such as DeepSeek and Llama. It exposes an OpenAI-compatible server implementing common OpenAI REST API endpoints such as the Chat, Completions and Responses API for text generation and inference, allowing existing clients and tools leveraging the OpenAI API to be reused simply by customizing the server URL. One of the main innovations of vLLM at the time it was initially released to the public is its paged attention mechanism addressing the critical bottleneck of key-value (KV) cache memory through its non-contiguous cache implementation, enabling significant efficiency and throughput gains in tasks such as text generation and inference.

    vLLM is an incubation project under LF AI & Data since October 2024 ensuring vendor-neutral governance.

    Follow me as I deploy my first locally hosted distilled variant of the DeepSeek-R1 model on a commodity Redmi Book 14 2024 laptop with vLLM in this exploratory lab. DeepSeek-R1 is a series of models trained via reinforcement learning available on Hugging Face.

    (more…)

    +

    +

    +

    +

    +

    +

    + ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Exploring the developer ecosystem on Huawei MateBook Pro

    +

    +

    +

    +

    +

    +

    Back in mid-2025, I stumbled upon the MateBook Pro in a Huawei flagship store in Shenzhen. The laptop caught my attention since it came pre-installed with Huawei’s very own HarmonyOS 5 operating system, otherwise known as HarmonyOS NEXT.

    Unlike previous HarmonyOS releases such as HarmonyOS 4 which were based on Android, specifically the Android Open Source Project (AOSP), HarmonyOS NEXT features a fully independent software stack ranging from Huawei’s proprietary HongMeng kernel, to the Kernel Abstraction Layer (KAL) and userspace based on the OpenHarmony project and HarmonyOS-native applications. It’s not common to find a laptop or PC in the wild which is pre-loaded with an operating system distinct from Windows or macOS so I decided to purchase it and give it a try.

    (more…)

    +

    +

    +

    +

    +

    +

    + ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Implementing GitOps practices with Argo CD

    +

    +

    +

    +

    +

    +

    The source code for this lab exercise is available on GitHub.

    GitOps is the practice of delivering infrastructure and applications in a declarative manner based on a single, authoritative source of truth. Oftentimes, the source of truth is defined within a Git repository, hence the term GitOps, though recent developments have started to decouple Git from GitOps with the source of truth defined in OCI artifacts instead in what is known as “Git-less GitOps”. Closely related is the concept of continuous delivery (CD) which is implemented in GitOps by continuously reconciling the actual state of the infrastructure and applications to match the desired configuration stored in the latest Git commit.

    Most GitOps implementations revolve around 2 CNCF Graduated projects:

    1. Flux: originally a WeaveWorks project; now spearheaded by ControlPlane
    2. Argo CD: actively developed on by startups and enterprise IT vendors such as Akuity and Red Hat

    This lab demonstrates the deployment of a sample podinfo application with GitOps using Argo CD deployed via the operator.

    (more…)

    +

    +

    +

    +

    +

    +

    + ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Configuring Rook with external Ceph storage

    +

    +

    +

    +

    +

    +

    Ceph is a distributed, scalable, fault-tolerant solution for provisioning block, file and object storage to applications.

    1. In Provisioning Ceph storage with cephadm, we provisioned a minimal Ceph cluster with cephadm and configured an RBD pool to provision block storage for applications
    2. In Running stateful workloads on Kubernetes with Rook Ceph, we deployed Ceph to Kubernetes with Rook using mostly default settings and provisioned block storage for MinIO with minimal effort

    While deploying Ceph to Kubernetes with Rook is simple and is sufficient for most Kubernetes use cases, certain scenarios may benefit from configuring Rook with an external Ceph cluster instead or as an additional option:

    1. Enables storage (Ceph) to scale independently of compute (Kubernetes)
    2. For large-scale deployments with high storage performance requirements best served by a dedicated Ceph cluster
    3. Sharing a Ceph cluster between Kubernetes clusters for multi-tenancy and maximizing storage utilization
    4. Deploying and managing Ceph separately from Kubernetes to separate administration duties among teams
    5. Providing additional storage capacity to Kubernetes workloads by deploying Ceph both internally and externally

    In the lab instructions to follow, we’ll configure a Ceph cluster with cephadm and configure Rook to connect to the external Ceph cluster.

    (more…)

    +

    +

    +

    +

    +

    +

    + ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +