Sergei Zobov | Blog

Advance your Robotics Startup: Monorepo and Functions

Sun, 07 Sep 2025 00:00:00 +0000

Background
Monorepo
Stop using Classes
- Use functions
Conclusion

Background

Over a decade, I worked in various small high-tech and robotics startups that often shared the same patterns: numerous repositories resembling “micro-services” architecture, OOP, and classes here and there.

None of these things is inherently bad, but they introduce unnecessary complexity that slows down small teams. Unfortunately, in robotics startups, it can kill your company.

Below, you’ll find a couple of low-hanging fruits that will help you advance your team.

Monorepo

“Microservices” became a trend when I was a junior developer. Almost every company I know brought it because it’s “cool”. For some reason, people consider the term “microservices” complete only if these “microservices” are split into different repositories. Additionally, people put mobile or frontend applications separately.

Cool, yes, but only if you’re a big company with many teams working on the same system.

If you’re not, a monorepo should be your go-to solution. Here is why:

Re-usage of packages

Suppose you have several private repositories and you want to reuse the code between them.

What are your options?

Set up Nexus/PyPI/npm/etc Then you have to build your package and upload it to the local registry. Are you done? Not yet: now you need to configure package managers everywhere to point to your local registry. Do not forget, that you can f*ck up configuration. Finally done? Not yet, because now you also need to figure out versioning.

The other option is using submodules, but you can easily find on the internet why it’s not the best idea.

Finally, you can use a monorepo!

Literally every package manager supports local imports. No more need to copy constants across several modules, just put them in the module and export them everywhere. Do you want something fancier, or does your codebase consist of too many different languages? Well, use buck2 or bazel (but more likely a few bash scripts will be enough in your case).

Re-usage of static files

Remember, I mentioned robotics? What do we like most? Big bags of triangles to represent our tools and robots? What do we like even more? Big ONNX files right into our repositories because we’re too lazy to properly set up S3-like storage.

Git was not meant to store big binary blobs, so companies made ad hoc solutions like GitLFS to deal with it. If you want to go this way now, you have to set it up everywhere you want to reuse your shiny big mesh!

With a monorepo, do it once, reuse it everywhere.

Suppose you have a bunch of repositories with C++ or Python packages. You have matured to the point where you want to use production technologies such as static analyzers and code formatters. You wrote your .clang-tidy, .clang-format or ruff.toml. What to do next? Of course, copy-paste them across all repositories! Now you want to change something and apply changes to all configurations, he-he…

Solution? Monorepo.

Put the configuration once in the root directory, and that’s it.

CI

well framed

Wanna run tests and set up a GitHub action for that? Don’t forget to set it up for all your repositories. Oh, now you want to trigger tests/builds in the dependent repositories when you push changes to the common one? If you have ever tried to do that, you know it’s not the easiest task.

Do you still want to do it, but don’t want to deal with all this hustle? Use a monorepo. All your dependencies are at hand. Moreover, most likely, you want to run all of your unit tests when you make a change to ensure you didn’t break other parts.

Integration tests get easier

Since I mentioned CI and tests.

An extra benefit of using a monorepo is simplified writing of integration tests. Put your mobile/frontend application in the same repository as your motion planner and enjoy catching bugs in displaying your metrics when the planner fails to plan all trajectories!

Docker

Monorepo doesn’t prevent you from using microservices. It just makes building them even easier. Instead of dealing with uploading Docker images to the in-house Docker registry and making sure you run the recent version, just build it right here.

More to this, you likely want to copy some of the files across multiple containers/images. With a monorepo, you are an enjoyer of COPY shared_src/....

Versioning

Versioning among different interdependent packages is hard and requires effort to make it right.

The first question you should ask yourself: Do I need per-component versioning at all?

Likely, the codebase represents your entire system. Monorepo perfectly matches it: you only need to put one tag when you do a release, and you’re good to go.

Ease transition

Imagine I convinced you to switch to the monorepo and merge all of your seven or eight others into one. How to make it simpler?

Do it iteratively; you don’t need to change all at once. Choose the first one: “repo A”. Make sure you have all in-work branches merged into “repo A”. After you merge “repo A” into the monorepo to merge these branches, it will require you to manually cherry-pick changes. Then, use git subtree add (without the --squash option to preserve the history).

Stop using Classes

Wherever I worked, I often saw people use classes everywhere in their code. I think it comes from the point that Java and C++ are widely studied in universities, and folks just get used to it.

I have no feelings about classes, but they often make writing tests much harder since the constructors become huge and objects require too many dependencies for initialization.

This is well described in the book “Working Effectively with Legacy Code” by Michael Feathers, which I highly recommend every software engineer to read.

We all know that tests are a must-have and they can drastically increase your development speed, but as with anything else, as hard as it is to do, we likely avoid them.

So, what’s the solution?

Use functions

Function, and as clean as possible.

It may sound obvious, but the beauty of a function is the interface: parameter and return type. No implicit dependencies as this-> or self.. Shit in - shit out.

On the contrary, several times I heard that people don’t like to split into functions, but having everything in one huge spaghetti, motivating that it’s “easier to grasp”. Well, god judge them (and to me, since I have a strong desire to punch in this case). The good thing about functions is that you can start integrating and testing them in places where it feels impossible.

Suppose you have a class MoneyCutter with a method execute and this method is around 400 loc. The class has a bunch of dependencies, such as ROS’ action clients, services and subscribers. Nobody ever dared to cover it with tests. You have to extend this method to support choosing the nearest money_disposal_box.

You have an option to just puke another 40 loc inside execute without any tests and be done. Let the person who will run the code deal with bugs.

The other option is to add a function: select_nearest_money_disposal_box(money_position, disposal_boxes) -> disposal_box_index and write a test for it. The choice is yours, but you know what will make the world a better place.

An illustrative example:

class MoneyCutter:
    def __init__(self, action_client, services, money_detector, disposal_boxes):
        self.action_client = action_client
        self.money_detector = money_detector
        self.disposal_boxes = disposal_boxes
        ...

    def execute(self):
        # 400 LOC
        disposal_box_id = select_nearest_disposal(money_position, disposal_boxes)

        pass

def select_nearest_disposal(money_position, boxes):
    return min(range(len(boxes)), key=lambda i: distance(money_position, boxes[i]))

def test_select_nearest_disposal():
    money = (0, 0); boxes = [(10,0),(1,1),(5,5)]
    assert select_nearest_disposal(money, boxes) == 1

Conclusion

tldr;

Default to a monorepo: pull services, apps, and shared assets into one place, keep linters/formatters at the root, build Docker right here, and tag the repo—not every component. Run the full test suite on each change. Push behavior into small, testable functions; keep classes thin and at the edges. Migrate steadily with git subtree add (no –squash) so history comes with you. Do these few things and your robotics team will spend less time wrangling toolchains and more time shipping real robots.

Do this and you’ll be alright.

When Recycling is a Challenge: Practical Tips to Make World Cleaner

Sun, 04 Aug 2024 00:00:00 +0000

Disclaimer
Background
Trash grabber
- Kids
Recycling collectors places
Switch to glass

Disclaimer

It is a non-technical blog post. I wrote it in English because it’s very easy to translate something from English nowadays.

Background

We never choose the country where we were born.

If you were lucky enough to be born in a country where it’s easy to re/upcycle, then it’s amazing!

I was not that lucky.

Unfortunately, recycling is not the easiest task in my country, and administrative people spend little (relatively) time cleaning the environment and land fields. It forced me to find ways on what I can do with it.

Since I was visiting my family and friends during my summer vacation, I compiled a list of things that I’ll describe in this article.

Trash grabber

One of my best investments was buying Trash Grabber, aka Reach Extender.

my trash grabber

I tried to collect litter with my hands before, but somehow, it raised the bar, so I quickly lost motivation. On the other hand, Trash Grabber made it fun and easy again! I use it in the following way: when I go outside in nature, I take it with me along with a trash bag. Then, I start to collect the litter around me. When all the trash bags are full, I try to find the nearest waste container or just put one under the tree so I can pick it up on my way home.

Easy and nice, right?

Well, it is. But do not do it very often. If you keep it seldom, it doesn’t become a heavy burden. Remember, getting burnt out is easy, so try to do it slowly but steadily.

One of the interesting side effects is that when people see you doing this with Trash Grabber, they start to get interested. A few people asked where to buy this thing. It may not be very sustainable, but you can prepare a paper printer QR code with a list of your local stores or online marketplaces where people can buy it.

Of course, just collecting and throwing waste in the bin has nothing to do with recycling. It’s not a sustainable solution. But it’s already a big thing. It’s much more psychologically easier to throw waste in nature if it’s already littered there.

I had a funny situation when I was collecting litter around two resting alkies, and they proudly proclaimed that they would collect all the waste they produced. It made me smile.

Kids

Kids like Trash Grabbers.

My nephew joins me every time I go out with Grabber, and he is around. This tool feels like something robotic. The pace of collecting slows a little, but I’m sure my nephew will overtake me with time. :)

My nephew and I collecting litter

My nephew and I collecting litter 2

Also, if you’re this type of person, you can make a Trash Grabber yourself with kids.

Recycling collectors places

Even if it’s not widely spread, there are already places where people can bring their waste for recycling. These could be containers for only glass, a specific type of plastic, or batteries.

Unfortunately, many such places are obscured and not easily found.

Let me give you a brief example from my experience.

I wanted to recycle glass bottles. I opened my home country’s most popular map application, Yandex.Maps, and tried to search for nearby glass recycling containers. The nearest one was about an hour’s walk from where I stayed. It’s not that far, honestly, but it requires a commitment from my side.

I decided to look in the search engine with the query something like “ glass recycling containers" (translated into my mother tongue). To my surprise, there were many results, and I quickly found a container that lasted about 5 minutes of walking. I was suspicious because who knows if the information was correct, but it's not a big deal to check it and, _voilà!_, it was right there. Moreover, there were four different containers to collect paper and various plastics.

I checked if I could quickly add my findings to the “Yandex.Map.” After clicking a few buttons, I submitted a new item to the map’s moderation. It was added in seven days!

Yandex.maps accepted my correction

Containers

Why did I tell you this story? Well, in my opinion, it will be easier to recycle if more people do it. On my next trip back home, I plan to collect more items and check if they all are easily foundable.

Switch to glass

This paragraph is very obvious, but I had never considered it before living in Germany. I was visiting my home country when it was hot there. I constantly need to stay hydrated. When I first walked into the grocery store, I had two options:

A more comfortable one – a plastic bottle of water.
Less comfortable one – a glass bottle of water.

The choice I made was, as you may already know, a glass bottle.

Glass is a material that people have been able to recycle, at least from the last century. Plastic, on the other hand, requires more technically developed infrastructure. Even if glass is not recycled and is left in the fields, it causes less damage to nature.

As soon as you start do it yourself people around you eventually will give it a try. Something very simple, but also powerful, when there are no better choices.

Debugging Machine Learning Model with Netron

Tue, 30 Apr 2024 00:00:00 +0000

Background
An issue
Bad guess
Working sample
Look at a structure
Source of the bug
Conclusion

Background

This article a part of a story I wrote in “Goodbye CUDA, Hello OpenVINO: Unexpected Benefits of Upgrading Python”.

During this upgrade, I faced an issue that initially felt like a show-stopper but became an easy-to-fix bug.

Let me tell you the details.

An issue

I had a machine learning model exported as ONNX model.

If I run an inference of this model using CUDA Execution Provider, it runs.

If I run an inference using OpenVINO Execution Provider it fails with the following error:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /home/onnxruntimedev/onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:53 onnxruntime::openvino_ep::OVExeNetwork onnxruntime::openvino_ep::OVCore::LoadNetwork(const string&, std::string&, ov::AnyMap&, std::string) [OpenVINO-EP]  Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_2_0Exception from src/inference/src/core.cpp:149:
Exception from src/frontends/onnx/frontend/src/core/graph_cache.cpp:25:
output/pos:0 node not found in graph cache

Bad guess

First, I tried to understand if I correctly configured an ExecutionProvider.

Since the error contains the word “cache,” I searched for a solution to clear this cache. I found that OpenVINOExecutionProvider has an option “cache_dir”, which sounded like a good candidate for fixing. I used this option and re-started an inference, and surprisingly, it worked, so I described it in the GitHub Issues.

Unfortunately, it was wrong. I misstructured the way parameters passed, and it turned out the inference was silently switching to the default CPUExecutionProvider. It was a big pity, so I needed to look for other solutions.

Working sample

Before continuing my search, I decided to check if this error was happening on all the files or on a particular one. Luckily, the following sample of the trained model I took worked without errors on OpenVINOExecutionProvider. That’s already something!

Look at a structure

The error tells us something is missing in the graph, so let’s look at the graph.

To visualize the underlying graph structure of our model, I used Netron: an open-source visualizer for neural networks.

Lucky me, Netron supports ONNX models as an input, so I started with visualizing the graph for the model that was rising an expection:

Then, I compered it to the working sample:

voilà!

The graph clearly shows that there are missing nodes. OpenVINOExecutionProvider gave me the legitimate exception!

Therefore, the fix should be done on the model side.

Source of the bug

It turned out the incomplete model was trained on the older version of the code, which used keras’s TerminateOnNan, which was softly terminating a training in case of NaN values encountered in the losses. When training data was malformed, it sometimes crashed on the first epoch and produced an incomplete model.

Conclusion

Solving the ONNX model issue with OpenVINO was tough but helpful.

At first, I thought the problem was about a cache setting, but that was wrong. The real issue was missing nodes in the model’s structure, which I discovered by using Netron to compare the broken model with a working one. This experience showed how important it is to use the right tools for debugging machine learning models.

RasPBX: connecting with my Grandma

Fri, 22 Mar 2024 00:00:00 +0000

Background
Disclaimer
The equation
The solution
The test
Other benefits

Background

I have a grandma.

She is already quite elderly and has some severe health issues.

I have lived abroad for two years and, due to some conditions, can’t easily travel to my home country. My grandma is an amazing woman, and I wanted to stay connected with her. The issue here is that she is unable to use smartphones and can only use push-button cell phones. Another issue is that it’s expensive for both of us to call using a traditional GSM network due to roaming.

This article will tell you how I overcame these issues.

Disclaimer

I’ve done some research on the available ready-to-use options. Somehow, I missed Microsoft’s options with Skype. I would have used this instead of tinkering with my solution if I had known it in advance. But after I implement mine, I’ll not switch to Skype since mine provides a few benefits that I’ll describe above.

The equation

(1) I have a SIM card with the local mobile number in my local county. The SIM card is needed to connect with another SIM card.

(2) I can call using my SIM card via smartphone.

(3) When I think of my smartphone, I see it as a small-sized computer with a built-in GSM modem.

If I substitute a smartphone from (2) via a small-sized commuter and a modem from (3), I can use the result to solve (1). With those thoughts, I started to look for solutions.

The solution

Once upon a time, I worked with Asterisk, so I knew that there are protocols that can connect phones to the Internet. By using the right keywords, I found RasPBX: the ready-to-use solution that one can roll out on various system-on-a-chip hardware, like RaspberryPI or BeagleBone Black.

Simply speaking, it’s a Raspbian image with Asterisk with the admin interface provided through FreePBX. More importantly, it has the drivers to connect a GSM model USB-dongle.

FreePBX interface

The list of supported dongles can be found here.

So, I ordered a second-hand Raspberry 2 and Huawei E169 dongle, which cost me approximately 50 euros. For the installation and configuration, I followed these steps from the documentation. The installation was bumpy because some steps from the documentation seemed missing. For example, I needed to create a user “asterisk” because the setup assumed it existed. Anyway, the installation took me only a short time.

RaspberryPI 2 and Huawei E169 connected

Moreover, a detailed manual is hosted here. It’s rich in images and can be an excellent place to find help.

Remember to connect your RaspberryPI 2 (if you’re using it) to the router via Ethernet cable since it doesn’t have a built-in WiFi module.

To simplify the network and security, I connected RaspberryPI to my VPN, so whenever I need to connect to Asterisk, I must first connect to my VPN.

That’s it.

The test

I installed ZoiPer on my phone to test the solution and called my mom first.

voilà! It worked! There was some delay in transmitting a sound, but we could clearly communicate (with some mental adjustment to always start speaking slowly to let the other side consider the delay).

I asked my mom to let my grandma know that I’d call so as not to scare her suddenly. Then I called her, and we finally could talk: from Berlin to a small village on the border with Kazakhstan with my SIM card physically in Saint Petersburg.

Zoiper interface for call

Looks like a classic phone calls software

Other benefits

Across the globe, mobile network providers do something that I don’t like: whenever a person is not using a SIM card for a while, it is sold to another person. Many of us (not except me) used their phone numbers as a second factor for authorization in such critical applications as mobile banks or e-government.

Also, my other family members still send me SMS with good words on my birthday. It’s nice to give them this ability.

I hope this article may help you too.

Goodbye CUDA, Hello OpenVINO: Unexpected Benefits of Upgrading Python

Sat, 03 Feb 2024 00:00:00 +0000

Background
Difficulties of upgrade
Metrics
Old CUDA vs new CPU
- Bigger model
OpenVINO
Conclusion

Background

At my current workplace I was tired of using an obsolete python 3.6 and decided to focus on upgrading python to something more recent, 3.10. My main reasoning was to improve dev-experience in using modern tooling and language feature. Also, bug fixes and safity patches.

It turned out, one of the side-effects of this upgrade was enable non-CUDA machine learning inference with a noticible perfomance boost.

This article will tell you about this side-effect and may also help you to convince your collegues on why upgrades are important.

Difficulties of upgrade

There is a bunch of problems I faced when I started the transition.

At this time we used quite an old Ubuntu 16.04 version and lead to some libraries were not available in public repos. The most critical ones were CUDA related libraries.

But why should I bother about CUDA at all? Well, because we’re using onnxruntime for machine learning inference. The version of this library is tightly bound to two things: the version of python and the version of CUDA toolkit. If I want to upgrade python I need to upgrade onnxruntime and inevitably CUDA.

At the time of the transition the only supported versions of ubuntu for CUDA were 20.04 and 22.04.

That’s a bummer.

I was puzzled if I should continue the transition, since one of the most important parts of our system is not available. The tough thoughts led me to a very dirty, but working solution.

I opened the instruction on installing CUDA toolit locally and noticed a pattern in how URL to downloading was built.

I built an URL for ubuntu 16.04 in the same manner and voilà! It worked!

After some back-and-forth (upgrade linux kernel and gcc) I was able to upgrade CUDA toolkit and install onnxruntime compatible with new python!

Show must go on!

Metrics

Let me give a little bit of context regarding metrics.

Whenever you’re doing anything regarding performance, you first need to find a reasonable way to measure this performance.

In our case, we had an exact place where the actual “work” was done. Measuring the time it takes us to run this code would be perfect because, in the end, it’s much easier for me to compare statistics from one place than trying to optimize several ones.

I was fortunate to have an exceptional engineer, Georges Dubus, who already brought to our project Open Telemtry stack.

The only thing left to me is to introduce a few “flags” to distinguish metrics from different setups.

Old CUDA vs new CPU

To this moment I had two setups: one running old python and one running new. More over, I made it in the way, so I can quickly switch between different Execution Providers.

After the first banch of benchmarks I was not going to belive it.

old vs new / GPU vs CPU (smaller is better)

The performance of a default CPU provider with newer version was roughly equal to the performance of CUDA/GPU provider with the older one.

Bigger model

I ran these benchmarks on a different setup, but the numbers were quite similar.

Sounds good, right? Well, yes, until I tested an inference of a bigger (deeper) machine learning model. Unfortunately, the performance of CPU Provider was not acceptable. So I begin the research on other options.

You may also ask me, if the performance of CUDA/GPU on a newer version was good enough, why didn’t I just chose this option. Unfortunately, as I mentioned above upgrade CUDA was quite a laborious work, required different steps executed in a very precise order. Therefore I didn’t try to fully avoid this option, but left it as a last resort.

OpenVINO

The hardware we using utilizes Intel CPU. Since after the python’s upgrade this CPU gives us a fair performance, can I squeze more from it?

The answer is yes. Intel made an open-source toolkit, called OpenVINO. The idea is very similar to CUDA: optimizing training and inference of machine learning models on vendor-specific hardware.

Lucky me, Intel already implemented an ONNX’s ExecutionProvider and even made pre-compiled wheels. The changes I need to implement to benchmark is adding a new provider “OpenVINOExecutionProvider” and executing pip install onnxruntime-openvino.

The only thing left to me was to run benchmarks and compare results.

Unfortunately, I don’t have nice plots to show here, but here is as table consiting the statistics for the same metric I described above.

provider	99th percentile	Average
CPUExecutionProvider	65ms	49ms
CUDAExecutionProvider	30ms	21ms
OpenVINOExecutionProvider	33ms	24ms

As you can see, the performance of OpenVINOExecutionProvider is roughly equal to CUDAExecutionProvider.

Conclusion

As you may already understood, I stick to the OpenVINO.

It was a bit of a journey and research, but in the end I brought an upgrade of the whole system and simplification of the requirements. OpenVINO runs on CPU, that means our product will run fully on CPU reducing the cost of the product! Something I didn’t expect when I started the upgrade.

Share feelings to improve communications

Fri, 15 Dec 2023 00:00:00 +0000

Disclaimer
Being a human
Example
My opinion

Disclaimer

A few days ago, I returned from a concise journey to my friend in Belgrade.

We spent approximately 90% of our time together speaking about different matters. So, this article will be the outcome of our discussion.

Non-technical, subjective and short.

Being a human

Working for almost a decade in technical companies, I noticed an essential and rather sad thing: people try to be as optimal as machines, showing no feelings. We try our best to speak concisely, delivering our intention fast and straight. As I see it, the problem is that we’re not machines, we’re human, and we should instead be proud of it and use it as an advantage.

Let me elaborate on it and provide an example.

Example

Last week, I woke up in a terrible mood. I had a sleepless night, and I felt broken.

When I opened our team’s chat, I noticed a few messages in direct messages. People mostly were asking about my opinion.

I started typing the answers, then realized: “Hold on, you’re definitely providing very negative feedback because you feel broke”.

I tried rewriting my message to be more objective, but it didn’t help.

Then, I decided to leave the message as is but add a disclaimer in the beginning:

10:03 AM

I just woke up in a bad mood

please divide any of my claims by a factor of 2 at least

and I then continued to share my opinion.

I don’t know precisely how my colleague reacted to it. What I know is that I at least provided this person with the knowledge of my being biased, not because of the topic this person brought up but because of totally unrelated stuff.

My opinion

Of course, it’s always better to be unbiased.

But I’m a human. I have bad and good days, which affects my decisions and communications. Most of the time, I’m pretty out of control of my mood. (Even tho I’m taking mood stabilizers, haha)

What I know I can do is:

identify my mood before I say shitty things
let people know my feelings before we start our communications
if I failed on both previous points, tell my feelings afterwards
be patient and empathetic to other people. They may also feel shitty

These are the steps that I’m trying to follow. I hope they will improve my communication.

I wish they could also help you.

Improving CI/CD experience with ChatGPT

Mon, 27 Nov 2023 00:00:00 +0000

Background
Reasons to use AI
Source of input
Results

Background

At my current company, before I joined, we had a build system based on Jenkins. I love Jenkins. It’s powerful, extensible, open-source and self-hosted.

But great power comes with significant responsibilities. Jenkins allows you to make bad decisions easily.

In our case, Jenkins’ Jobs were fully defined in UI. Why is it wrong? Well:

It’s impossible to track it in Version Control System such as git.
It’s easy to screw up. If you accidentally change a step in UI, you can only spot it through your naked eyes.
It’s hard to get a whole picture. In our case, it required to scroll a screen 4-6 times to see a whole file.
It’s hard to reuse code. (Still possible tho)
Restarting the pipeline from the beginning or specific stage is impossible.

I decided to address all the mentioned problems. This article will tell you how I moved to pipelines defined by Jenkinsfile with the help of ChatGPT.

Reasons to use AI

There are no particular reasons to do it.

It’s an easy but tedious task to structure Jenkinsfile.

Due to my laziness, I was seeking the most effortless approach. Since I’m paying for ChatGPT, I said, “OK, AI is not so bad at summarizing texts”. In this case, the problem is summarizing: UI is a wordy source, and Jenkisfile is a short summary.

Source of input

When I was implementing the migration, it was impossible to send pics to ChatGPT. Even if possible, I wonder if I can treat it as good input. Jenkins jobs we defined in UI are stored somewhere, so I started searching for the source of the job’s definitions.

I was lucky to quickly find the exact question on StackOverflow. The proposed request to the HTTP handler didn’t work for some reason, so I just ran find <jenkins_dir> -name config.xml -type f and immediately found the configuration for the pipeline defined in XML.

We’ve got a source! It’s not perfect since it’s a colossal XML, but who cares? Let’s feed it now into GPT!

Results

After some back and forth, ChatGPT was able to generate something that I wanted. It didn’t go deep, but I didn’t ask it for. I wanted some skeleton or boilerplate code that I could then extend.

Of course, I had to go through every command and thoroughly read the documentation to understand what was happening in this script. But the cool thing is that I know what to search for.

In my case, it was a perfect example of using ChatGPT. I already had a working job that I wanted to get. The thing that I should have in the end should be a drop-in replacement.

Few more bonuses:

It allows me to ask about specific parts or share the names of the plugins it used.
I can feed other XML files.
It takes less mental power for me to edit something rather than to create from scratch.

Here is the reduced, high-level script I’ve got from ChatGPT so you can see that there is nothing specific, just boilerplate lines:

pipeline {
    agent { label 'built-in' }
    parameters {
        string(name: 'NAME', defaultValue: '',
               description: 'Feature name')
        booleanParam(name: 'UPLOAD_BUILD', defaultValue: false,
                     description: 'Upload build to cloud')
        string(name: 'BRANCH', defaultValue: 'feature_build',
               description: 'Git branch to build')
    }

    environment {
        GIT_URL = 'git@github.com:...'
    }

    stages {
        stage('Clone Repository') {
            steps {
                checkout([$class: 'GitSCM',
                          branches: [[name: "${params.BRANCH}"]],
                          doGenerateSubmoduleConfigurations: false,
                          extensions: [],
                          submoduleCfg: [],
                          userRemoteConfigs: [[url: "${env.GIT_URL}"]]])
            }
        }

        stage('Run Versions Script') {
            steps {
                script {
                    ...
                }
            }
        }

        stage('Trigger Builds') {
            steps {
                script {
                    build(job: '...', parameters: [string(name: 'VERSION',
                          value: "${env.VERSION}")], wait: false)
                    build(job: '...', parameters: [string(name: 'VERSION',
                          value: "${env.VERSION}")], wait: false)
                    build(job: '...', parameters: [string(name: 'VERSION',
                          value: "${env.VERSION}")], wait: false)
                }
            }
        }

        stage('Copy Artifacts') {
            steps {
                script {
                    copyArtifacts(projectName: '...', filter: '...',
                                  target: '/builds/')
                }
            }
        }

        stage('Run Post-Build Operations') {
            steps {
                script {
                    if (fileExists('./post_build.sh')) {
                        sh './post_build.sh'
                    }

                    if(params.UPLOAD_BUILD) {
                        ...
                    }
                }
            }
        }
    }
}

Reverse engineering binary protocol from dev console

Tue, 14 Nov 2023 00:00:00 +0000

Prerequisites
Background
Promising beginning
Broken API
Reverse engineering
Result

Prerequisites

I wrote this article with the assumption that the reader has some prior knowledge about JavaScript, bytes, and Presentation layer. If you don’t have it, you may still benefit from reading, but it may be too boring. With prior knowledge, it is still boring, but less. ;)

Background

As I describe in my previous blog post I wrote a telegram bot to simplify cards creation in Anki.

In this article, I’ll describe some technical issues I faced in integrating with Anki, particularly reverse engineering their communication protocol.

Promising beginning

Anki doesn’t provide a public HTTP API. At least on a date of writing this article.

When I started my integration by spending a few minutes in dev-console, I quickly realized that Ankiweb is working by simply sending GET/POST requests with forms. Sounds easy, right? Speaking honestly, it was easy. The implementation took me an hour, and the client was fully functional.

Everything worked smoothly, but then it suddenly broke.

Broken API

If there is no public API, there are no promises. I should have expected it.

My bot stopped adding cards to Anki and could not even log in. I spent some time figuring it out and understood that the API has changed, and Anki is no longer simply sending forms.

I still strongly desired to add words to my decks, so my next Journey began.

Reverse engineering

Ankiweb was still using HTTP requests and cookies to preserve the authentification information. It means I should be able to reproduce what Ankiweb’s front end is doing.

My first issue was that I could not see the response to requests in the dev console. It simply showed me:

Ouch! It was a bit confusing because I was sure it should somehow exchange the data. Otherwise, how does it work?

Then, I looked into the login request’s payload. I also set a password to start from a very particular string so it will be easy to spot it in logs.

payload of the login request

Nothing very suspicious except for”” and “)” symbols. It felt like some binary data was transferred around, but what could it be?

I started to look at the source code of the web page. Obviously, it was minified and obfuscated. Literally, just a bunch of unreadable JavaScript expressions joined in one infinitely long string.

{if(r=V[n.charCodeAt(c)],r===void 0)switch(n[c]){case"=":a=0;case`
`:case"\r":case"	":case" ":continue;default:throw Error("invalid base64 string.")}switch(a){case 0:i=r,a=1;break;case 1:s[t++]=i<<2|(r&48)>>4,i=r,a=2;break;case 2:s[t++]=(i&15)<<4|(r&60)>>2,i=r,a=3;break;case 3:s[t++]=(i&3)<<6|r,a=0;break}}if(a==1)throw Error("invalid base64 string.");return s.subarray(0,t)},enc(n){let e="",s=0,t,a=0;for(let r=0;r<n.length;r++)switch(t=n[r],s){case 0:e+=S[t>>2],a=(t&3)<<4,s=1;break;case 1:e+=S[a|t>>4],a=(t&15)<<2,s=2;break;case 2:e+=S[a|t>>6],e+=S[t&63],s=0;break}return s&&(e+=S[a],e+="=",s==1&&(e+="=")),e}}

It is a pile of junk, but can we make use of it?

Browsers are our friend in reverse engineering frontends. For example, Chromium provides a pretty-printing option for minified JavaScript. After pretty-printing it, we get something like this:

    if (a.status === 403)
        return window.location.href = "/account/login",
        new Uint8Array;
    if (!a.ok) {
        let c = a.statusText;
        try {
            c = await a.text()
        } catch {}
        throw Oe(a.status, c)
    }

Still hard to read since it’s minified, but it gives us two advantages:

It’s easy to find where requests are made since we can directly search for substrings.
The most essential point: it’s now easy to set debugger breakpoints and check all values in runtime.

I set a debugger right into the code where the request happened, and it led me to another request, which was not displayed in the dev console for some reason. It turned out there are many requests that start with the /svc/ prefix. The particularly interesting one is here:

So, I moved into this function.

async function u(n, e, s, t) {
    const a = e.toBinary()
      , r = await ne(n, a, t);
    return s.fromBinary(r)
}

What’s happening here is definitely something important for request forming:

e contains our login and password information in such format: {username: 'by***', password: 'pass***'}
Then, using some logic, the code makes a binary array from it and sends it to the server.
In case of success, it receives a bunch of bytes and decodes them back to the JavaScript object.

Now, my problem is figuring out the “logic” behind encoding and decoding data between requests and responses from the server. First, I looked into the e.toBinary() function since encoding happened here. By jumping back and forth, I got to the following object:

0: 
    T: 9
    jsonName: "username"
    kind: "scalar"
    localName: "username"
    name: "username"
    no: 1
    packed: false
    repeated: false
    [[Prototype]]: Object
1: 
    {no: 2, name: 'password', kind: 'scalar', T: 9, localName: 'password', …}
    

On the bright side, it already gives some insight; on the other side, how this object is formed is still unclear. I started to dig deeper into the sibling of the toBinary() function: fromBinary(). Let’s now look closely at this function:

    fromBinary(e, s) {
        const t = this.getType()
          , a = t.runtime.bin
          , r = a.makeReadOptions(s);
        return a.readMessage(this, r.readerFactory(e), e.byteLength, r),
        this
    }

Where the parameter “e” contains the binary data received from the server. I also checked that the output of this function is a nicely formed object.

I was lucky to get a jackpot from the first guess, and I printed the t.runtime object:

> t.runtime
< {syntax: 'proto3', json: {…}, bin: {…}, util: {…}, makeMessageType: ƒ, …}

You may wonder why I was so happy to see this object.

Because of the “proto3” strings, which immediately clicked in my mind: it’s a protobuf! 🔥

And now the object I mentioned a few lines above makes perfect sense: it’s a message representation from the protobuf! Armed with this knowledge, I started to craft the protobuf messages from the object I got from the dev console.

A few words about protobuf: I already mentioned it one of my old articles. It’s a binary serialization protocol, likely OSI level 6, and compiler, which is widely used and a cornerstone of gRPC technology. You describe the message structure, and then it automatically creates a code to handle it.

So, I formed a first login message and transferred it from my code. Ta-da! It worked. We cracked this nut.

Unfortunately, I quickly realised something was still missing in my messages. They work nicely, but sometimes, the server responds with an error for some requests.

I looked more closely at the data I de-serialized from the server and noticed some peculiar values for IDs. For example, deck_id has a value -143234, which clearly sounded like a bug, probably interger overflow. Again, I was lucky to solve this issue from the first guess: I initially used the int32 format for int values, probably because it’s unclear from the object in the console. I switched to uint64, and all issues were resolved.

Result

The resulting implementation of the API client can be found here.

The interface became more consistent, so I’m very much thankful to Anki’s maintainers for this change. It was an exciting journey, and I didn’t regret any time spent on it.

Towards better natural language learning

Sun, 22 Oct 2023 00:00:00 +0000

Background
Anki
Source of words
Messengers to the rescue
Conclusion

Background

A while ago I started to learn new languages. In the beginning, it was English, then Japanese and now German.

Since then, I have always tried to optimize this process using technologies. And since we now have a powerful computer in our pockets, I was looking for software to extend my vocabulary in a new language.

That is how I came to Anki.

Anki

Briefly, it’s an application or even a framework, that allows you to create flashcards and with a remarkable spaced repetition algorithm learn effectively (at least for me) new things.

I’m sure everyone has their best way to learn new words in a new language, but for me, a classical method of just writing words and trying to learn them all at a time simply doesn’t work. At first, it felt like I had learned, but all the new knowledge swiftly faded.

I found Anki a fantastic tool to solve the problem mentioned above:

First, you learn new words in batches. The default setting is only 10 new words per day for a deck of words and 100 cards to repeat. The numbers are configurable, but it worked perfectly, so I don’t feel overwhelmed.
Second, the spaced repetition algorithm helps words stick to your mind. The first time, you need to repeat a word in 10 minutes, then in 3 days and after a few more times, it can be more than 1 year.

heatmap of my repetition in Anki

The next puzzle we need to solve is filling Anki with the words we want to learn!

Source of words

Many say grammar is important. I say only vocabulary can let you speak freely.

What could be the best source of new words? For me, the answer is simple: books.

But the problem with books is that reading the whole book in a language you’re not fluent in can be challenging. It’s also hard for your brain without seeing progress, and it can be even more difficult with books. You can start with books for children, but if you’re not a kid, it could be not very attractive.

For me, the answer was Japanese comics called manga. The beauty of manga is that it’s made for all ages. Also, many genres and beautiful drawings made it very attractive to me. My choice was “Dorohedoro” translated into German. It has everything I need: an exciting story, beautiful drawings and a tons of informal lexicon.

me reading manga

Messengers to the rescue

Now we have two most important pieces: Anki and books.

The problem we still have is that adding words from the book to Anki is difficult.

You can use the Anki mobile application to add new words, but it was not the easiest solution for me. When learning a new language, it’s crucial to make the process as simple as possible. Otherwise, it’s easy to get frustrated and quit.

What can solve this issue? Something I use every day.

The answer is messengers, particularly Telegram. I spent some time designing and developing a telegram bot Anker.

The UI of this interface is simple: you choose a language and a deck and send it a word to translate. Then, it creates a card from the word and translation. After syncing the app with the cloud, it appears as this nicely-looking card:

Now, we have all the pieces to improve our vocabulary drastically.

Conclusion

I stopped learning Japanese as soon as I decided to relocate to Germany. I mostly forgot all the grammar I knew, but I still can understand the words I learned using Anki.

After my weekly German lesson, I add new words to my learning deck. Anki made it scalable for me. It gives me confidence that I don’t waste my time in the learn-and-forget loop but have slow, steady progress.

I understand that the method I describe in this article can only fit some people, but I hope you can try it and it can work for you.

Black box testing for Android application

Thu, 12 Oct 2023 00:00:00 +0000

Background
Available options
Maestro
Downsides
Conclusion
- Disclaimer
- Acknowledgement

Background

A few months ago, I asked a question on LinkedIn:

Unfortunately, it didn’t attract much attention, and I still wondered. In my current company, we use an Android application as a frontend to our complex backend system. This Android application is the first and only user-facing, so we needed a tool to ensure its quality.

Before I joined, it was almost always manually tested, but the problem of this approach could be more scalable. We needed automated tests but wanted to avoid getting into the Kotlin/Java code to write them.

Our objective was: we need a tool to write automated tests which don’t require Android/Java/Kotlin knowledge.

Available options

We did initial research on the possible solution. I’m unsure if it’s a complete list, but these are the things we’ve found:

screenshot-tests-for-android – a snapshot testing testing tool, but a little unflexible if UI of your application changes often. Also, it requires integration in the source code of the application.
BrowserStack – a cloud-based platform to run tests on real or virtual devices. It seems like a powerful instrument, but vendor-locking, recurrent subscription and complexity make us hesitant to try it. On its own, it doesn’t provide a test framework but supports running many.
Espresso – Android UI test framework. Again, this requires integration with the source code and Kotlin/Java knowledge.
Appium – as stated in the docs: “is an open-source project and ecosystem of related software, designed to facilitate UI automation of many app platforms”. It is a complex UI test framework that can run on different platforms: physical, virtual devices or BrowserStack. We tried to run a setup, but it didn’t feel easy and required many integration steps.
Maestro – The first sentence in the docstrings stated: “is the simplest and most effective mobile UI testing framework.”, which made me a little sceptical, but after we tried it turned out to be our choice. I’ll describe it in detail in the next section.

Maestro

It is a rather new and relatively obscure tool. I found the first mentioning on hackernews dates Aug 31, 2022.

We chose this tool over the other since:

Simple Language

The language of writing tests is simple DSL written in YAML files. The language is simple but capable of expressing complex flows. This is the first example we wrote for our testing:

appId: com.mirai.app
---
- launchApp
- extendedWaitUntil:
    visible: "Skill Overview"
    timeout: 70000 # ms (eq 70 sec)
- assertVisible:
    text: "Add new skill"
    enabled: true

You can see that it’s very descriptive. All the commands cover what I usually do when I use our application. Also, it’s possible to express loops and importing from other files. You may say it’s relatively too simple, but in my personal feelings, complicated language leads to cumbersome and often hard-to-maintain tests.

Decoupled from the Application

Maestro setup knows nothing about your application.

All app-specific logic is expressed only in the tests themselves. Maestro only starts the application as a user does it and then runs a flow of commands against it. You can change the language your mobile app is written in, change your build system, and do whatever you want – as long as the interface is the same, your maestro tests will stay as they are.

Decoupled from the Hardware

You can run your tests against a physical device or emulator.

For our setup, I built a Docker Image based on docker-android.

Speaking frankly, it required a small tests update to increase the timings since the application in the emulator works slowly, but I would say it’s expected.

However, the apparent benefit of wiring up these tools is that it is a perfect setup for your CI pipeline.

The hardware devices are excellent, but they could be more scalable, or you need to pay cloud providers like the mentioned BrowserStack. But with Docker, you can run as many instances of the simulator as could fit into your CI server.

Click-to-test UI

Maestro includes a browser-based UI where you can click and select the elements of your app and convert them to the tests.

I consider it a killer feature since it simplifies the test writing workflow.

Readability

The file with UI tests can overgrow, and needs help comprehending. Maestro support includes code from other files via the runFlow command. You can use label to make tests more self-descriptive with human-readable text.

Downsides

Maestro is a very new tool and is under active development.

Some features need to be documented, and you need to upgrade your installation occasionally.

While I was writing this blog post, I realized that the “labels” I mentioned in the previous topic were reverted and are no longer available. I may predict there could be bugs and breaking changes.

Conclusion

We were pleased to find such a powerful and accessible tool despite all the downsides mentioned. Emulating devices with Docker and running automated tests gave us many opportunities to reduce manual testing time. Now, we can run as many devices as we need and pinpoint bugs earlier.

Disclaimer

I’m not affiliated with any of the mentioned projects.

Acknowledgement

This post is based on my work together with Maria Matyushenko.

Sergei Zobov | Blog

Advance your Robotics Startup: Monorepo and Functions

Background

Monorepo

Re-usage of packages

Re-usage of static files

Share configuration

CI

Integration tests get easier

Docker

Versioning

Ease transition

Stop using Classes

Use functions

Conclusion

When Recycling is a Challenge: Practical Tips to Make World Cleaner

Disclaimer

Background

Trash grabber

Kids

Recycling collectors places

Switch to glass

Debugging Machine Learning Model with Netron

Background

An issue

Bad guess

Working sample

Look at a structure

Source of the bug

Conclusion

RasPBX: connecting with my Grandma

Background

Disclaimer

The equation

The solution

The test

Other benefits

Goodbye CUDA, Hello OpenVINO: Unexpected Benefits of Upgrading Python

Background

Difficulties of upgrade

Metrics

Old CUDA vs new CPU

Bigger model

OpenVINO

Conclusion

Share feelings to improve communications

Disclaimer

Being a human

Example

My opinion

Improving CI/CD experience with ChatGPT

Background

Reasons to use AI

Source of input

Results

Reverse engineering binary protocol from dev console

Prerequisites

Background

Promising beginning

Broken API

Reverse engineering

Result

Towards better natural language learning

Background

Anki

Source of words

Messengers to the rescue

Conclusion

Black box testing for Android application

Background

Available options

Maestro

Simple Language

Decoupled from the Application

Decoupled from the Hardware

Click-to-test UI

Readability

Downsides

Conclusion

Disclaimer