Tiny struggles

Implementing SARM on your VLA dataset in practice

Sun, 14 Dec 2025 00:00:00 +0000

1. Motivation: use big video dataset optimally for VLA training

In the first part of this series, I explained what is SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation and how it can be used with a challenge such as Stanford Behavior Challenge (1200h of demonstrations over 50 diverse long horizon tasks).

To sum up: Our main model for the robot (VLA) is trained on short windows (chunks) of data for which it predicts actions. Our SARM model is used to estimate progress within an episode and evaluates windows. We want to prioritize training on trajectory segments where the robot made meaningful progress toward task completion.

In this post I will explain how I actually implemented this in practice. The code is now open on github. The core of the implementation follows closely the original paper.

We will cover the following key areas:

The design of the model Sequential Multimodal Architecture that utilizes a Global Anchor Frame.
Data input shape and preparation
Using the model for scoring the episodes
Visual Validation of the predicted progress against our Stage-Aware Ground Truth.

2. The SARM Model Implementation

See the source here.

The model tackles a dual prediction problem: determining which stage of a task is being performed (classification) and how much progress has been made within that stage (regression).

SARM provides a principled approach to stage-aware reward modeling by:

Leveraging pretrained vision models (CLIP) for robust visual understanding
Fusing multimodal information through Transformers
Making hierarchical predictions (stage → progress within stage)
Handling variable-length sequences efficiently

Architecture Overview

This architecture is particularly well-suited for tasks that have clear sequential structure and require fine-grained progress estimation within each stage.

The SARM model follows a three-part design:

Encoders - Process multimodal inputs (visual and proprioceptive)
Shared Backbone - A Transformer that fuses information across time and modalities
Dual Heads - Separate outputs for stage classification and progress regression

Where the symbols are:

B - batch size
N - sequence size (multiple frames of images/data - more on that later)

┌────────────────────────────────────────────────────────────────────┐│                         INPUT LAYER                                ││                                                                    ││  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              ││  │ Image Frames │  │ Joint States │  │  Task Index  │              ││  │  (B,N,3,     │  │  (B,N,256)   │  │     (B,)     │              ││  │   224,224)   │  │              │  │              │              ││  └──────┬───────┘  └───────┬──────┘  └────────┬─────┘              │└─────────┼─────────────────-┼──────────────────┼────────────────────┘          │                  │                  │          ▼                  ▼                  ▼┌─────────────────────────────────────────────────────────────────────┐│                      ENCODER LAYER                                  ││                                                                     ││  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               ││  │  CLIP (ViT)  │  │  LayerNorm   │  │  Embedding   │               ││  │   [Frozen]   │  │      +       │  │    Layer     │               ││  │      ↓       │  │   Linear     │  │              │               ││  │   Linear     │  │              │  │              │               ││  │  Projection  │  │  Projection  │  │              │               ││  │              │  │              │  │              │               ││  │ (512→768)    │  │ (256→768)    │  │ (50→768)     │               ││  └──────┬───────┘  └──────┬───────┘  └───────┬──────┘               ││         │                 │                  │                      ││         │   Visual        │   State          │   Task               ││         │   Embeddings    │   Embeddings     │   Embedding          ││         │   (B,N,768)     │   (B,N,768)      │   (B,1,768)          ││         └─────────┬───────┴──────────────────┘                      │└───────────────────┼─────────────────────────────────────────────────┘                    │                    ▼          ┌─────────────────┐          │  Element-wise   │          │      Sum        │          │                 │          │  Visual + State │          │    + Task       │          └────────┬────────┘                   │                   ▼          ┌─────────────────┐          │  Add Positional │          │  Bias to Frame 0│          └────────┬────────┘                   │                   │  Combined Embeddings                   │  (B,N,768)                   ▼┌────────────────────────────────────────────────────────────────────┐│                   TRANSFORMER BACKBONE                             ││                                                                    ││  ┌───────────────────────────────────────────────────────────────┐ ││  │  Transformer Encoder (8 layers)                               │ ││  │                                                               │ ││  │  ┌─────────────────────────────────────────────────────────┐  │ ││  │  │  Multi-Head Self-Attention (12 heads)                   │  │ ││  │  │  d_model = 768,                                         │  │ ││  │  │  Dropout = 0.1                                          │  │ ││  │  └─────────────────────────────────────────────────────────┘  │ ││  │                           ×8                                  │ ││  └───────────────────────────────────────────────────────────────┘ ││                                                                    ││                   (with padding mask support)                      │└──────────────────────────────┬─────────────────────────────────────┘                               │                               │  Aggregated Features                               │  (B,N,768)                               ▼┌─────────────────────────────────────────────────────────────────────┐│                         OUTPUT HEADS                                ││                                                                     ││         ┌──────────────────────┴──────────────-────────┐            ││         │                                              │            ││         ▼                                              ▼            ││  ┌─────────────────┐                          ┌─────────────────┐   ││  │  Stage Head     │                          │  Subtask Head   │   ││  │  (Classifier)   │                          │  (Regressor)    │   ││  │                 │                          │                 │   ││  │  Linear(768→512)│         ┌────────────────┤  Concat:        │   ││  │      ReLU       │         │                │  - Features(768)│   ││  │   Dropout(0.1)  │         │                │  - Logits(100)  │   ││  │  Linear(512→100)│─────────┘                │                 │   ││  │                 │                          │ Linear(868→512) │   ││  │  Stage Logits   │                          │      ReLU       │   ││  │  (B,N,100)      │                          │   Dropout(0.1)  │   ││  └─────────────────┘                          │   Linear(512→1) │   ││                                               │     Sigmoid     │   ││                                               │                 │   ││                                               │ Scalar Progress │   ││                                               │    (B,N)        │   ││                                               └─────────────────┘   │└─────────────────────────────────────────────────────────────────────┘

Data Flow Explained

1. Input Processing

The model accepts three types of inputs for each sequence:

Image Frames (B, N, 3, 224, 224): A batch of N RGB images per sequence
Joint States (B, N, D_state): Robot proprioceptive information (joint angles, positions, etc.)
Task Index (B,): An integer identifying which task is being performed

Where B is the batch size and N is the maximum sequence length, the actual data can be shorter and then we pad it.

For every prediction, our model processes a sequence of frames, deliberately structured to provide maximum temporal context:

Global Anchor Frame: The first frame of the episode is included in every sequence. This is a crucial engineering choice for long-horizon tasks, as it gives the Transformer a global, unchanging reference point for the task’s initial state.
Subsampled Context Frames: Several preceding frames are included to capture recent history.
Current Frame: The frame for which the progress prediction is required.

2. Encoding Stage

Each modality is processed through its own encoder:

Visual Encoding:

Images are flattened from (B, N, 3, 224, 224) to (B*N, 3, 224, 224)
Passed through a frozen CLIP ViT-B/32 model to extract visual features
CLIP outputs 512-dimensional features per image
Features are projected to the model dimension (768) via a linear layer
Reshaped back to (B, N, 768)

State Encoding:

Joint states are normalized using LayerNorm
Projected from dimension 256 to 768 via a linear layer
Output: (B, N, 768)

Task Encoding:

Task index is converted to a learned embedding vector
The embedding is replicated across the sequence: (B,) → (B, 1, 768)
This embedding is broadcast and added to all timesteps

3. Multimodal Fusion

The three encoded representations are combined:

input_embeddings = visual_embeddings + state_embeddings + task_embedding

Additionally, a learned positional bias is added only to the first frame:

input_embeddings[:, 0, :] += positional_bias

This creates a unified representation (B, N, 768) that contains information from all modalities.

4. Transformer Backbone

The combined embeddings are processed through an 8-layer Transformer encoder:

Architecture: Standard Transformer encoder with 12 attention heads
Dimensions: 768-dimensional hidden states, 3072-dimensional feedforward layers
Padding Support: The model accepts an optional padding mask (B, N) where True indicates padded positions
Output: Aggregated features (B, N, 768) that capture temporal and multimodal dependencies

5. Dual Output Heads

The model produces two types of predictions:

Stage Head (Classification):

Takes the aggregated features (B, N, 768)
Passes through: Linear(768→512) → ReLU → Dropout → Linear(512→100)
Outputs stage logits (B, N, 100) representing 100 possible task stages:
- 100 is a maximum number of task stages supported, in practice the stages will be task dependent
Trained with cross-entropy loss

Subtask Head (Regression):

Concatenates aggregated features with stage logits: [features, stage_logits] → (B, N, 868)
This conditioning allows progress estimation to be stage-aware
Passes through: Linear(868→512) → ReLU → Dropout → Linear(512→1) → Sigmoid
Outputs scalar progress (B, N) in the range [0, 1]
Trained with MSE loss

Loss Computation

The SARMWithLoss wrapper handles training:

Masking: Only non-padded positions are included in loss calculation
Stage Loss: Cross-entropy between predicted logits and ground truth stage labels
Progress Loss: MSE between predicted progress and ground truth progress values
Total Loss: Weighted sum of both losses (default weights: 1.0 each)

total_loss = (stage_loss_weight × stage_loss) + (progress_loss_weight × progress_loss)

Including the loss calculation within the model wrapper made the training code simpler and more standard.

Key Design Decisions

These decisions follow the original SARM paper:

Why freeze CLIP?CLIP is pretrained on massive image-text datasets and provides robust visual features. Freezing it prevents overfitting on smaller robotics datasets and reduces computational cost.
Why condition subtask head on stage predictions?We estimate progress within a stage, not the whole episode, so it’s stage dependent.
Why add positional bias only to the first frame?The first frame often contains important context about the initial state. The positional bias helps the model distinguish the starting point from subsequent frames. Supposedly such anchoring is very effective for video models.
Why use variable-length sequences with padding?

We use Rewind Augmentation when we sometimes generate longer sequences that ‘mess up’ progress on purpose by replaying older frames in the reverse order. Because of that we need to handle sequences of varied length. This augmentation is critical for the model to learn how undoing progress looks like.

3. Complex data preparation

The data for SARM has to be prepared in a very particular way.

The core of the sampling is implemented in the custom dataloaders here.

Temporal Sampling Strategy

SARM doesn’t sample frames uniformly. Instead, it uses a sophisticated sampling strategy designed to provide temporal context:

def prepare_indices(ep_first_frame_idx, idx, skip_count=30,                    default_length=8, rewind_prob=0.05):    # Sample backwards in time with skip_count intervals    indices = [idx - i * skip_count for i in range(default_length)]        # Always include the first frame of the episode    indices.append(ep_first_frame_idx)        # Reverse so time flows forward    indices = list(reversed(indices))        # 5% chance: add "rewound" frames for temporal augmentation    if random() < rewind_prob:        num_extra = random.integers(2, 5)        indices += [indices[-1 - i] for i in range(1, num_extra + 1)]        return indices

This creates sequences with the following structure:

Sequence construction (skip_count=30, ~1 second at 30 FPS):┌───────┬───────┬───────┬───────┬───────┬───────┬───────┬───────┬───────┐│Frame 0│ t-7s  │ t-6s  │ t-5s  │ t-4s  │ t-3s  │ t-2s  │ t-1s  │  t    ││(start)│       │       │       │       │       │       │       │(curr) │└───────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┘With 5% probability, add rewind frames:┌───────┬───────┬─────────────┬───────┬───────┬───────┐│...    │  t    │ t-1s (again)│ t-2s  │ t-3s  │(curr) │└───────┴───────┴─────────────┴───────┴───────┴───────┘

Why this design?

Always anchor to episode start: Frame 0 provides consistent context about initial conditions
Uniform temporal spacing: 1-second intervals capture motion patterns without redundancy
Rewind augmentation: Teaches the model temporal reversibility and robustness
Future context avoided: Model only sees past and present, not future frames

Delta Timestamps Pattern

The sampling strategy is complemented by a clever timestamping scheme:

DELTA_TIMESTAMPS = [HIGH_NEGATIVE_TIMEDELTA] + [-7 + i for i in range(8)]# Results in: [1e6, -7, -6, -5, -4, -3, -2, -1, 0]

When applied to current timestamp t:

Frame 0: Gets timestamp ≈ -∞ (approximated as episode start)
Frames 1-7: Get timestamps [t-7, t-6, …, t-1]
Frame 8: Gets timestamp t (current)

This ensures consistent temporal windows regardless of where you are in the episode.My custom dataset SARMDataset uses a dataset provided by the BEHAVIOR codebase under the hood that allows specifying delta timestamps for more efficient sampling.

Variable-Length Sequence Handling

Real episodes have variable lengths, and sequences can have different numbers of frames. SARM handles this with:

def collate_fn(batch):    # Find max sequence length in batch    max_length = max(sample["sequence_length"] for sample in batch)        # Pad all sequences to max_length    batched_images = torch.zeros(batch_size, max_length, C, H, W)    batched_padding_mask = torch.ones(batch_size, max_length, dtype=torch.bool)        for i, sample in enumerate(batch):        seq_len = sample["sequence_length"]        batched_images[i, :seq_len] = sample["images"]        batched_padding_mask[i, :seq_len] = False  # False = valid, True = padding

The padding mask is then passed to the Transformer to ensure padded positions don’t contribute to attention or loss:

Example batch with lengths [9, 11, 13, 9]:Padded to max_length=13:┌─────────────┬─────────────┬─────────────┬─────────────┐│ Seq 1 (9)   │ Seq 2 (11)  │ Seq 3 (13)  │ Seq 4 (9)   │├─────────────┼─────────────┼─────────────┼─────────────┤│ [V][V]...[V]│ [V][V]...[V]│ [V][V]...[V]│ [V][V]...[V]││ [P][P][P][P]│ [P][P]      │             │ [P][P][P][P]│└─────────────┴─────────────┴─────────────┴─────────────┘  V = Valid token  P = Padding token (masked out)

Inference Sampling Strategy

To use SARM for VLA training, we need to run our original video dataset through SARM.

But that dataset was huge to begin with! But we don’t need to evaluate every frame (with its proceeding sequence).

With 5-second sampling at 30 FPS, I evaluate only 1 out of every 150 frames (5s × 30 FPS), reducing computational cost by 150×.

I implemented a following dataloader (source).

Why jitter?

Jitter prevents the model from overfitting to fixed timestamps and produces more robust progress estimates by sampling at slightly varied intervals rather than exact multiples of 5 seconds.

See INFERENCE_README for more details on the inference.

Translating the progress to VLA training weights

Additionally I implement the progress mapping to the weights following the SARM paper (equations 8-9):- Computes progress deltas r̂ᵢ = φ(t+Δ) - φ(t)- Uses running statistics (μ, σ) to normalize- Applies linear ramp between (μ - 2σ) and (μ + 2σ)- Optionally uses threshold κ for decisive weighting

See the code in weight utils.

4. Training & Results

This implementation of SARM was multi-task, however since there was so much data, I decided that it would be easier to evaluate and visualize it on a single task first.

Having 200 episodes for each task, I divided the data into the following sets:

“train_episodes”: 1-90,
“val_episodes”: 91-105,
“test_episodes”: 106-200

In general performance on the validation set wasn’t the best indicator of actual model performance when I analyzed it on the test dataset. Training for more steps was helpful.

For training details see the config and the training script. The 10k-step snapshot has been trained on a single RTX5090 over several hours. The model is also compatible with training on MPS. The key bottleneck was the dataset access and video processing.

Visualizations & analysis

I performed detailed analysis on how well the models were predicting the progress here.

Ground Truth

First, it’s important how the ‘ground truth’ data looks like, here is a visualization:

Based on the data annotations and the stage statistics I was able to generate our ‘Ground truth’ of progress. It was also a useful sanity check if the ground truth data looks right, e.g. having negative progress in the ground truth data would mean that there were bugs. We were only adding ’negative progress’ through the Rewind augmentation later on.

Comparing models vs ground truth and each other

Model checkpoint comparison vs ‘ground truth’ on a sample of episodes:

The first 3 episodes were in the training data and the 2 last ones weren’t present.You can see here that the yellow (10k steps) model is better fitted to the data in the training set.

Understanding the bias of the model

So the model wasn’t perfect, what type of mistakes was it making?

Overall, the model was leaning towards underestimating the progress. And the key problem was from predicting wrong stage number.

Applicability and Limitations

The caveat here that the task 8 was multimodal, the stages could be done in variable order, breaking the fundamental assumption of the fix stage order in SARM. Visualizations for a task fitting SARM assumptions would look better.

The SARM Assumption: Stage-Aware modeling assumes a generally linear path through semantic checkpoints (Stage 1 $\rightarrow$ Stage 2 $\rightarrow$ Stage 3…).
The Failure Case: Multimodal Progress: If a task allows for subtasks to be completed in an arbitrary order (e.g., “Tidy up the room”), the progress estimation becomes inherently multimodal, and our regression model, forced to average these possibilities, loses accuracy.

Handling different sequences of stages in demonstrations

Removing outliers

If the majority of demonstrations are done in a consistent way, then we can remove the outlier demonstrations that create confusion. (Annotations to generate ground truth are enough to blacklist such episodes).

Subtask splitting

The SARM model implemented here can handle multiple tasks.

Therefore, if a task can be done using different sequences of stages, we can transform it into set of related tasks with different demonstration variations.

If there are multiple different ways represented in similar proportions, e.g. ‘pick up toy 1’ then ‘pick up toy 2’, and the reverse, we can change into two tasks pick_up_toys_1_2, pick_up_toys_2_1.

Equal sampling

We can also decide not use SARM for such tasks.

SARM in Behavior challenge

For the BEHAVIOR challenge specifically, we were very time constrained, and we ended up not having enough time to apply the model for the final checkpoint training, additionally, only about 30% of tasks fulfilled the fixed stage ordering for SARM.

We performed quick fine tuning with weighted sampling earlier on a subset of data (not based on SARM), but it was difficult to see if it was actually helpful (eval in general was pretty challenging).

Despite these challenges, SARM remains a promising approach for datasets with proper stage annotations and sequential task structure. Our analysis on the 30% of tasks with fixed stage ordering showed the model could accurately track progress. For the remaining tasks, the subtask splitting approach outlined above could make SARM applicable, potentially enabling more efficient VLA training through intelligent data selection across the full dataset.

Improve Your VLA training: Using Reward Models to Filter Best Training Data

Wed, 29 Oct 2025 00:00:00 +0000

In recent years, there has been huge progress in AI for robotic manipulation. VLAs such as $\pi_0$ ($\pi_{0.5}$) are the current state of the art, but it’s still difficult to get good performance on complex, long-horizon tasks.

Exactly the type of tasks as in the Stanford BEHAVIOR Challenge that I have been working on (as a group effort) for the last couple of weeks. The challenge consists of 50 full-length household tasks, with 200 demonstrations each. And the tasks are long, 1200h, often a single demonstration is longer than 10 minutes.

For imitation learning - having a great demonstration dataset is a key differentiator. How to improve the dataset? If it’s small, you can always record more demos, even though it’s expensive and time-consuming. But what if it’s already big? In BEHAVIOR challenge, the dataset is already huge, in original quality it’s about 2TB and training on its entirety could be taking weeks.

It’s hard to assess the quality of the demonstrations as there are 12000h of it. Maybe there’s a better way to train than just doing behavior cloning on the entirety of the dataset?

One of the approaches we wanted to try as a group is using some RL methods to augment the VLA training. This is what SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation addresses. The paper uses the challenging task of T-shirt folding (long horizon task somewhat equivalent to the BEHAVIOR tasks) to demonstrate why standard fine-tuning fails and presents a clever, two-part framework to solve it:

train a reward model that assigns high rewards to samples with high progress (using two staged approach)
use the reward models in the fine tuning of a target model (incorporating it in the loss function), it makes the model to pay attention more to demonstrations with high progress

This post is a deep dive into what makes this paper work, focusing on the tricky parts:

Quality issues of large datasets fine-tuning on a 200-hour dataset can result in 0% success.
How SARM re-thinks “progress” labeling to create a stable reward signal
The “Rewind” data augmentation trick that’s essential for learning “failure”
Why this reward scheme is valuable: The paper’s two-part evaluation, and why one test is for accuracy and the other is for usefulness
How this method differs from the typical use of reward models in RL.

as well as translating the ideas into practical implementation on a large dataset.

1. The Problem: “Garbage Data” Kills $\pi_0$ Fine-Tuning

The SARM Paper’s goal is to fine-tune $\pi_0$ on T-shirt folding, a “long-horizon, contact-rich manipulation task” involving deformable objects. The authors break down the task’s complexity:

Easy: “Picking the shirt from a box” (~5 seconds).
Medium: “Folding a T-shirt from a flattened state” (30-60 seconds).
Hard: “Folding from a crumpled state” (1-3 minutes).

The “Hard” task is the real challenge, as it requires long-term planning and handling uncertainty. A look at the paper’s “dense annotations” shows the “fold” stage alone consists of 5-7 distinct sub-tasks, like “grab near side and fold,” “rotate the tshirt 90 deg,” etc.

The standard approach is to fine-tune $\pi_0$ on a large dataset using standard Behavior Cloning (BC). The authors did this with a 200-hour dataset, “BC-All,” and the results were stark:

Easy Task: 100% success.
Medium Task: 8% success.
Hard Task: 0% success .

The problem isn’t $\pi_0$. It’s that a 200-hour dataset of a 3-minute task is inevitably “noisy.” It is filled with “suboptimal trajectories”—pauses, fumbled grasps, and inefficient recovery motions. Standard BC learns from this “garbage data” just as much as it learns from expert motions, leading to a confused policy that imitates the failures.

2. The Solution: Weighted Fine-Tuning (Repurposing the reward model)

Reward-Aligned Behavior Cloning (RA-BC) introduced in the paper tries to address this problem by learning a reward model, that is then used to weigh the loss during the VLA fine tuning (BC).

How Reward Models are Usually Used in RL

In RL, the agent tries to select actions that maximize rewards.Typically, a reward model is used online. In a framework like the ReWiND paper, an agent (trained with an RL algorithm like SAC) interacts with the world, and the reward model provides a live, dense reward signal to guide its learning. The agent then learns from this feedback over millions of interactions.

SARM’s “Offline” Approach

The SARM paper is primarily about imitation learning (RL is briefly explored in the appendix). Instead of RL, it uses its reward model offline as a one-time data filter. The RA-BC (reward aligned behavior cloning) framework is a “weighted” fine-tuning of $\pi_0$.

Here’s how it works:

A reward model (SARM) is trained to be an expert at scoring progress.
This reward model scores every single clip in the entire 200-hour noisy dataset.
A “progress delta” is calculated for each clip: $\hat{r} = progress_{end} - progress_{start}$
This $\hat{r}$ is mapped to a weight $w$ for the BC loss function between 0 and 1.
The final $\pi_0$ fine-tuning is done with this weighted loss: ${L}_{RA\cdot BC}(\theta) = \frac{\sum w_i \cdot \text{loss}(i)}{\sum w_i}$.

This forces the $\pi_0$ model to only learn from the high-quality, high-progress segments and completely ignore the “garbage data”. The success of this entire method now hinges on one thing: the quality of the reward model.

3. SARM: Building a Reward Model That Actually Works

The paper’s core contribution is its two-ingredient recipe for building a robust reward model.

Ingredient 1: “Stage-Aware” Labeling (To Learn Progress)

The paper argues that to describe progress, one needs to identify a task stage and the progress within that stage for multi-stage, varied-trajectory tasks.

Prior work often relies on frame indices as labels (…). While thismay suffice for short tasks with fixed duration, such as “pick up the cup,” it fails for tasks like“fold the T-shirt,” where trajectories vary greatly, task duration is not fixed, and motion sequencesdiffer across demonstrations. For example, in T-shirt folding, the flattening phase may require moreor fewer motions depending on shirt placement or fabric configuration, yet frame-based labelingonly reflects elapsed time. As a result, identical task states (e.g., a fully flattened shirt) can receiveprogress values ranging from 0.2 to 0.8, introducing severe label noise that harms reward modellearning and downstream policy training.

The SARM Solution:
1. Define semantic Stages (e.g., “Grab,” “Flatten,” “Fold”) .
2. Calculate the dataset-wide average time proportion for each stage (e.g., “Flatten” takes 25% of the total time, on average).
3. This creates fixed progress checkpoints. “Grab” is always the 0.0 $\rightarrow$ 0.1 window, and “Flatten” is always 0.1 $\rightarrow$ 0.35, etc..
4. Labels are then generated by interpolating between these fixed checkpoints.

Now, the “fully flat shirt” state always gets a label of 0.35 (or some other consistent value). This provides a stable, consistent signal for the model to learn.

Note: this stage + progress within the stage idea is also used for adding a ‘simple system 2’ for a VLA system that helps the model not get lost at long horizon tasks. It’s especially helpful when the state isn’t fully captured and observations are not Markovian (e.g. next stage might look the same as the previous stage, but we need to do different things now).

Ingredient 2: “Rewind Augmentation” (To Learn Failure)

In our imitation learning dataset, we don’t want to teach the robot how to make mistakes on purpose. We might want to teach it how to recover from problems, but not how to introduce them in the first place.The training demos are all (mostly) successful, but some accidental failures (and recoveries) might sneak in.

How can we avoid accidentally teaching the robot to make mistakes?

The Solution: The paper adopts a key technique from the ReWiND paper: “rewind augmentation”.
How it Works: During training, the system takes a successful video clip (e.g., frames 1-10) and appends frames from earlier in the clip in reverse order (e.g., frames 1-10 are followed by frames 8, 7, 6).
The Result: The model is explicitly trained to predict decreasing progress scores for this “rewound” section. This teaches the model to recognize and penalize actions that undo progress, which is “essential for building reward models that generalize to real-world policies”.

4. Demystifying the Evaluation: “Accuracy” vs. “Usefulness”

How is the SARM model assessed? The model is evaluated on its own as well as in the RA-BC setup (compared to simple fine tuning or other reward models).

The authors evaluate their SARM model with two different methods in Table 1, and they are not the same . It’s a test of “Accuracy” vs. “Usefulness”.

Test 1: Loss on human demonstrations (Model Accuracy)

Question: Is our model accurate at predicting the ground-truth labels on clean, unseen human demos?
Method: A standard regression test. It measures the Mean Squared Error (MSE) between the model’s predicted progress and the true stage-aware label.
Result: A low score is better. This just proves the model learned what it was told.

Test 2: “Rollout of robot policy” (Model Usefulness)

Question: Are the model’s scores useful for judging messy, real-world robot rollouts? Can it tell success from failure?
Method: This is a classification test, and it has an extra step.
1. A robot policy is rolled out (interacts with the environment to perform the task) to create multiple episodes that are then labeled as Successful (SE), Partially Successful (PSE) or Failure (FE).
2. The same SARM model (trained only on clean demos) generates its normal progress scores for a messy, out-of-distribution robot video.
3. The evaluators (not the model) apply a ruleset to these scores: “IF (final_score > 0.8) AND (avg_progress_last_third > 0.6) THEN classify as ‘Success’”.
4. The final score ($\rho$) measures how well these predicted labels match the true human labels (SE, PSE, FE).
Why this matters: Real-life interactions with the environment are much messier than the ‘ideal demos’, so the data from such rollout (demonstration) is Out of Distribution (very different) from what was encountered in the demonstration. This test proves the model is robust and its scores are meaningful for filtering real-world failures (and it would also make it significantly more useful for online RL where OOD problems are deadly). The paper shows the baseline ReWiND reward model fails this “usefulness” test, which is why the RA-BC-ReWIND policy ultimately failed.

5. How to Apply SARM to Your Own Dataset (A 4-Step Guide)

This section will explain how the ideas from the paper could be applied to a real-world example, like the BEHAVIOR challenge dataset (50 tasks, 200 episodes/task, all annotated).

Step 1: Define Canonical Stages (Per-Task)

This is the most critical step. For each of your 50 tasks, define its unique, semantic stages.

Task 12: (Preparing Lunch Box): “Put both apple halves, the club sandwich, and the chocolate chip cookie from the chopping board on the kitchen countertop into the packing box on the countertop. Then take the bottle of tea out of the refrigerator, put it into the same box, and close the refrigerator when you’re done.”
Task 7 (Picking Up Toys): “Put all the toys in the child’s room - the three board games (two on the bed and one on the table), the two jigsaw puzzles on the table, and the tennis ball on the table - inside the toy box on the table in the child’s room.”

The tasks have text descriptions as well as detailed annotations with time frames. Each annotation can correspond to a separate stage.

Annotation file example from the behavior dataset:

{    "task_name": "preparing lunch box",    "data_folder": "",    "meta_data": {        "task_duration": 7876,        "valid_duration": [            90,            7966        ]    },    "skill_annotation": [        {            "skill_idx": 0,            "skill_id": [                1            ],            "skill_description": [                "move to"            ],            "object_id": [                [                    "packing_box_210"                ]            ],            "manipulating_object_id": [],            "spatial_prefix": [],            "frame_duration": [                90,                343            ],            "mp_ef": [],            "skill_type": [                "navigation"            ]        },        {            "skill_idx": 1,            "skill_id": [                2            ],            "skill_description": [                "pick up from"            ],            "object_id": [                [                    "packing_box_210",                    "countertop_kelker_0"                ]            ],            "manipulating_object_id": [                "packing_box_210"            ],            "spatial_prefix": [],            "frame_duration": [                344,                619            ],            "mp_ef": [],            "skill_type": [                "uncoordinated"            ]        },        {            "skill_idx": 2,            "skill_id": [                1            ],            "skill_description": [                "move to"            ],            "object_id": [                [                    "burner_mjvqii_0"                ]            ],            "manipulating_object_id": [],            "spatial_prefix": [],            "frame_duration": [                619,                733            ],            "mp_ef": [],            "skill_type": [                "navigation"            ]        },        {            "skill_idx": 3,            "skill_id": [                91            ],            "skill_description": [                "place on next to"            ],            "object_id": [                [                    "packing_box_210",                    "burner_mjvqii_0",                    "chopping_board_211"                ]            ],            "manipulating_object_id": [                "packing_box_210"            ],            "spatial_prefix": [                [                    "",                    "",                    "right"                ]            ],            "frame_duration": [                734,                817            ],            "mp_ef": [],            "skill_type": [                "uncoordinated"            ]        },       ...    ]}

Step 2: Generate Stage-Aware “Ground-Truth” Labels (Per-Task)

This step must be performed separately for each of your 50 tasks.

Isolate Task: Get the 200 annotated episodes for one task (e.g., “Preparing Lunch Box”).
Calculate Proportions: Find the average temporal proportion of each stage only for that task’s 200 episodes.
Create Checkpoints: Create the fixed progress checkpoints for that task (e.g., add_the_chocolate_cookie ends at 0.15, take_the_bottle_of_tea ends at 0.7).
Interpolate: Generate the new, consistent ground-truth labels for those 200 episodes by linearly interpolating between those task-specific checkpoints.
Repeat: Go to the next task and repeat.

At the end, you have 10,000 episodes with new, consistent labels, where the progress bar is calibrated correctly for its specific task.

Step 3: Train One Multi-Task SARM Model

You do not need to train 50 separate reward models.

Instead, you train one, single, multi-task SARM model on your full 10,000-episode dataset. The key is to make this model task-aware by feeding it the task instruction (just like a VLA).

Your model’s input would be:

Video Frames
task_id

The model (with its dual stage/progress heads) will learn to predict the correct stages and progress conditional on the task instruction.

“Rewind Augmentation” is the special sauce. This is where the model learns how failures look like. Otherwise one could just use the original “Ground Truth” and it would be simpler.

Step 4: Integrating the reward model with VLA training

In this section I will describe how the SARM can be used for VLA training in practice.

The paper mentions changing the loss function to incorporate the weight based on the reward for a given window for a training sample, but I had an insight that weighing the gradients for learning can be achieved in asimpler way, by using weighted random sampler to over/under represent the parts of the training set.

So we will use the SARM model to Pre-Generate Weights once & use the weights for the weighted sampler for VLA training.

This section assumes that you are training your VLA on action chunks (e.g., of length 30) and that you have support for a Weighted Sampler that can take per frame weights.

We will create a list of weights for each frame of the dataset assuming a fixed chunk size.

For each task:

Slide your 30 step chunk window (from $t$ to $t+30$).
For each chunk, get the progress delta by feeding the model the task instruction:
- $P_{start} = \phi(d_t, task\_id)$
- $P_{end} = \phi(d_{t+30}, task\_id)$
- $\hat{r} = P_{end} - P_{start}$
Map this task-specific $\hat{r}$ to a final, non-negative weight $w$ (e.g., if $\hat{r} \le 0$, $w=0$; else $w=1$)
Store $w$ in your global list of weights.

After this, you will have a single, massive list of weights, perfectly aligned with your training chunks. You can then feed this into a WeightedRandomSampler to train your single, multi-task VLA (like $\pi_0$) using the standard BC loss.

No changes to the loss function necessary, pretty much all the changes are in the data loader.

Next steps & conclusions

I don’t have results yet when it comes to using the SARM for the BEHAVIOR challenge yet, fundamentally we don’t even really know how good the dataset is already.

Taking better advantage of the datasets we already have is very smart, especially when the demonstrations come from human operators of varied skill, so I expect ideas such as SARM to be relevant for a long time.

The obvious next step having a good Reward Model would be to try to use it with RL, especially if a simulator is present. The paper described an effective way to do it with a diffusion model, but it’s not yet clear to me how to effectively translate it to RL training of a complex model such as $\pi_{0.5}$.

Robotics Hackathon in Bimanual Manipulation in Munich

Sun, 05 Oct 2025 00:00:00 +0000

How a LinkedIn Post Led Me to a Munich Basement with Millions of Euros Worth of Robotics Equipment

My LinkedIn feed has become a stream of robotics content over the past few months. As someone diving deep into AI robotics after years in ML/AI/RL, I’ve been deliberately connecting with people pushing the boundaries of the field. So when Nicolas Keller’s post about Munich being “the world’s best place to build robots” appeared in my feed, it immediately got my attention.

A bimanual manipulation hackathon in Munich, organized in just three weeks. How cool is that?

Here’s what they promised:

Hands-on with dual-cobot humanoid upper-body setups, 2 Franka Emika Pandas, 2 depth cameras, 1 rgb, RTX5090 in each station
Teleoperation with Meta Quest VR headsets
Contributing to the MINGA research paper (with co-author potential)
Collaborating in Munich’s growing robotics ecosystem

I had three weeks to apply, get accepted, and arrange travel during what turned out to be Oktoberfest (the Lederhosen on the robot in the picture should have been a hint! It just made the stay extra expensive).

I have recently missed the LeRobot’s worldwide hackathon and wanted to jump on the opportunity. The prospect of 30+ dual-arm robot setups, high-end GPUs, industry mentors and meeting other people passionate about robotic manipulation made it a no-brainer decision for me.

The hackathon experience

Most hackathons are short and they only involve software. Hardware Hackathons are much more rare, especially where hardware is provided and high end.

The organizers promised a lot - and I have to say that they over-delivered, even though they operated on a very short timeline! 3 weeks!

The hackathon wasn’t perfect; we hit some technical issues with the provided codebase and the robot controllers.The lab space (KI Fabrik in the Deutsches Museum) was full of amazing robots, powerful workstations, and a 3D workshop, but it also had its downsides - it was hot, humid and you could get trapped there due to the limited number of keys!

The schedule was focused on building: after one day of setup and tutorials, it was essentially “09:00–open ended — Building time” for six straight days. The main communication happened on Discord.

The setup was industrial-grade: 30+ Franka Emika Panda dual-arm configurations for about 40 participants. Each setup came with Meta Quest headsets running custom teleoperation software. There was a full workshop with 3D printing capabilities for custom grippers. The compute power was serious—workstations with the latest hardware that most of us don’t have access to.

The initial goal of the organizers was to attract local students (mostly from TUM), but the hackathon was just too attractive.

The organizers ran the selection process based on a Typeform where you had to justify your presence (CV, motivation, experience) and the final mix of people contained: PhD researchers, startup founders, industry engineers, and ambitious students. There was a significant number of people who traveled from other countries. Everyone wanted to be there.

Industry engagement and realistic use cases

The real differentiator was the industry backing. BMW and Siemens provided realistic challenges to be solved, explained the details, provided physical materials and sponsored prizes.

Additionally, there were helpful lectures and mentoring from Nvidia, Hugging Face LeRobot and KIT (a big German university).

When Sunday’s final presentations came, BMW and Siemens employes showed up to judge the results personally. This wasn’t academic theory - teams were working on problems that companies actually need solved, with the decision-makers accessible throughout and present for the final outcomes.

The main organizers were TUM and Poke & Wiggle. I was very impressed with both. TUM showed great support for students and entrepreneurship. Poke&Wiggle people pulled everything together from the technical side. They were staying late and even hosted some participants coming from abroad in their homes! They were testing their own software stack while building what they claimed would become the largest public bimanual manipulation dataset.

My strategy & Learning

Probably my favorite thing about this hackathon was that it was extremely collaborative. Yes, people tried to win, but I was able to learn both from my team as well as from others.

Team formation & collaboration

I came to the event without knowing anyone and needed to form a team. It was actually a pretty common experience, many people didn’t know who to pair with. I have been in setups like this before and that experience helped.

To form a good team you want the best people, but also it’s really hard to assess people very quickly and people who seem great at the start might not be able to give the best performance. E.g. they might not be fully available, you might not get along very well, etc.

So my strategy was to talk to most people, see how they think and how experienced they are. I was selecting for getting along, enthusiasm and general intelligence. It was less important to me that someone was inexperienced as long as they were energetic and open minded.

We weren’t the most effective or best organized, but we really enjoyed our time together, made good progress and learned a lot. Our team also shifted a bit during the 7 days (one person got sick and we adopted another one).

We used a WhatsApp group to share resources, set up a GitHub repo for shared scripts and shared some notes on a Google doc.

Our strategy was as follows:

get familiar with the teleoperation
trying out all available tasks and assess the task feasibility for the human operators
train and deploy the models ASAP to test the pipeline (yes, we detected bugs and further limitations)
pursue the most promising tasks, refine the dataset collection and experiment with the models for good performance

Learnings

We didn’t manage to win any categories. I think that my group was more focused on learning and experimentation, instead of purely competing to win.

Some takeaways:

If the task can’t be done by the human, the robot won’t be able to do it
Training loss during model training is not a good predictor of the real life performance
Robot safety mechanisms were critical to avoid breaking the robot
Evaluation in real life is risky and some simulation setup would be helpful
End-effector control with inverse kinematics was often causing the robots to get stuck due to joint limits; it required special care during teleoperation for the demos so that the actual policy wouldn’t block the robot
Two arms are much harder than one
Dataset quality matters a lot (recovery examples, non-Markovian states are confusing, noise/operators in the setup)

What we tested

3 different task setups with hundred+ demonstrations each
variations in model training (steps, parameters, models, action space) and dataset selection (recovery episodes ratio, bad episodes)
different control modes (delta and absolute)
we could only try actions in the EE space, the joint space controllers weren’t working correctly
SMOLVLA and ACT models from Lerobot libraries
performed experiments for generalization and resilience (e.g. messing with cameras was making robots much less effective) we also visualized attention maps for the ACT Models

Here is an example of an attention map:

(Kudos to physical AI Interpretability Repo - the author was there during the event and helped us a bit with the setup).

Misc

I got a pretty good feel for teleoperation in VR (it’s hard though!)
I managed to get the arms to crash with each other and got my robots stuck countless times
I spent some time setting up a simulation environment with Panda in MuJoCo and playing with Isaac Sim (approach abandoned in the end)
I read several papers recommended by other participants
I wasn’t able to try Pi0/Pi0.5 as the ready snapshots are for the same type of robot (Panda), but in a very different action space and we didn’t have time/resources for fine-tuning from scratch (80GB+ GPU memory required)

What I wished I could do:

try out Groot / Pi0.5
simulation, sim-to-real, and RL fine-tuning for the trained VLA (a simple VLA would be perfect!)

Posts describing the experiences of some of my teammates: @Artur and @Andrea.

We Need More of This

After seven days of intense collaboration in a basement, we didn’t revolutionize the future of robotics, but we all learned a lot and everyone came back home more experienced and inspired.

It was a great event!

And I want to see more events like this in Europe, because the future of robotics doesn’t have to happen in SFO (or China)

Europe has the ingredients for world-class robotics innovation. We have strong engineering talent and strong reasons to invest (aging population)! I live in Poland now, a place that produces some of the world’s best software engineers, but the innovation is lacking. I talked to two universities in Warsaw and they don’t really innovate or even follow the current state of the art yet for embodied intelligence.

And it’s a shame, because with libraries such as LeRobot, open hardware, and open-source simulation engines, the space is now much more accessible.

I was very impressed with TUM and many of the students. I would like to support the ecosystem in Poland and Europe. Munich proved it’s possible. Let me know if you would like to help!

Next steps

I left the event pretty drained, but also very excited! JI am still following up on the various threads I started during the hackathon. I also started to look at another exciting challenge that is focused on household tasks in the simulation.

Currently I’m especially interested in the approaches combining VLAs with RL, like the ones outlined in the SimpleVLA paper and would love to participate in more hardware hackathons, ideally combining simulation and real world learning.

Robotics Training Experiments: What Worked, What Didn't, and What Surprised Me

Sun, 17 Aug 2025 00:00:00 +0000

Part 2 of a series on practical robotics experimentation. See part1.

After building my simulation environment, the real learning began. I threw multiple approaches at the cube-to-bin task: Soft Actor-Critic (SAC) reinforcement learning, behavior cloning from demonstrations, and modern imitation learning with LeRobot. Each taught me something different about the practical realities of robot learning.

Here’s what I discovered—including some findings that challenged my assumptions and revealed the gap between academic ideals and development reality.

You can find the training code here.

Why SAC? Guidance from ChatGPT and the Literature

When it came time to choose a reinforcement learning algorithm for manipulation, I turned to ChatGPT for guidance on what would work well for the precise, continuous control that robotic manipulation demands.

The Algorithm Landscape:

PPO: Great for many RL tasks, but can struggle with the precise, smooth actions needed for manipulation
DDPG: Deterministic policy gradients, but known for training instability
TD3: Improved version of DDPG with better stability
SAC: Soft Actor-Critic with entropy regularization

Why SAC Emerged as the Right Choice:Through my conversations with ChatGPT and reading the literature, SAC stood out for manipulation tasks:

Smooth, continuous actions: Essential for robotic control
Sample efficiency: Better than policy gradient methods like PPO
Stability: More reliable than DDPG/TD3 for continuous control
Exploration: The entropy term helps discover diverse manipulation strategies

The Manipulation Sweet Spot:Robotic manipulation requires policies that can be both exploratory (to discover grasping strategies) and precise (for fine motor control). SAC’s soft policy approach naturally balances these needs.

My background in ML and deep generative modeling helped me understand why the entropy regularization was important, but ChatGPT’s domain-specific guidance really pointed me toward SAC as the practical choice for this type of problem.

This background influenced my entire approach to the experiments that followed.

You’re absolutely right! I made the same mistake again. Let me fix that section:

The SAC Experiments: Dense vs. Sparse Rewards

The Reward Shaping Struggle:I didn’t initially know about HER. I first spent several days training and tweaking reward shaping—trying sparse and dense methods, experimenting with penalizing for missing the goal on every step versus not penalizing. When I saw poor training success and weird behaviors from my trained agents, it made me investigate alternative strategies.

SAC with Dense Rewards: The Winner

For dense rewards, I shaped the reward function around manipulation primitives:

# Simplified reward structureapproach_reward = -distance_to_cubegrasp_reward = gripper_closure_bonus if near_cube else 0lift_reward = cube_height_above_table * lift_multiplierplacement_reward = -distance_cube_to_bin if cube_grasped else 0success_reward = 100 if cube_in_bin else 0

This approach worked reliably. The policy learned smooth approach trajectories, consistent grasping, and coordinated placement motions. Training was stable and converged predictably.

SAC + HER with Sparse Rewards: A Later Discovery

Why HER Seemed Appealing:When I discovered HER, it promised a more disciplined approach to the reward engineering problem I’d been struggling with: just define success and let the algorithm figure out the intermediate steps. The idea is elegant—when an episode fails to reach the intended goal, HER treats it as if it was trying to reach wherever it actually ended up. This creates synthetic “success” experiences from every failed attempt.

The HER Insight:Instead of manually crafting rewards for “getting closer to the cube,” HER automatically discovers that reaching certain intermediate positions is useful for eventually reaching the final goal. It’s like learning to play chess by treating every game as a success for reaching whatever position you actually achieved.

For robotic manipulation, this seemed perfect:

No manual reward engineering: Just define task success
Automatic curriculum: Learn easier goals (approach) before harder ones (grasp+place)
Sample efficiency: Every episode contributes training signal

The Implementation Reality:But first, I had to figure out how to actually implement HER, which wasn’t obvious. The documentation and examples didn’t make it clear how to structure the environment properly.

The Goal Environment Challenge:HER requires a “goal-conditioned” environment where observations include both the current state and a target goal. See my implementation example.

The HER Training Setup:Once you have the goal environment structure:

from stable_baselines3 import HerReplayBuffer, SACmodel = SAC(    "MultiInputPolicy",  # Required for dict observations    env,    replay_buffer_class=HerReplayBuffer,    replay_buffer_kwargs=dict(        n_sampled_goal=4,        goal_selection_strategy='future',    ),    verbose=1)

The honest truth about my experience:

Learning progress was painfully slow—each step of improvement took much longer compared to dense rewards
I never actually had the patience to let it finish training—the progress was so glacial that I always gave up and went back to dense rewards
The few times I did wait longer, performance was unstable

Was I doing it right? It’s entirely possible that I wasn’t implementing the GoalEnv correctly, or that my hardware was too slow for HER to show its benefits. The implementation wasn’t obvious from the documentation, and debugging goal-conditioned environments is more complex than standard ones.

For the patient (or well-equipped): If you have more patience than I did—or faster hardware—this goal-conditioned setup might work well for you. But life is short and iteration speed matters.

Lesson learned: Academic papers often focus on sample efficiency, but wall-clock time and learning progress rate matter enormously for practical development. Sometimes the “less elegant” dense reward approach is the right engineering choice.

The SAC Training Setup: Real Implementation Details

Here’s what my actual SAC training pipeline looked like (full implementation in scripts/train_sac.py):

Multi-Environment Training:

# Create vectorized environments for faster trainingvec_env = make_vec_env(    create_single_env,    n_envs=num_envs,  # I used 2-6 parallel environments    vec_env_cls=SubprocVecEnv,    env_kwargs={"task": task},)vec_env = VecTransposeImage(vec_env)vec_env = VecNormalize(vec_env, norm_obs=True, norm_reward=False, clip_obs=10.0)

Model Configuration:

model = SAC(    policy="MultiInputPolicy",    env=vec_env,    learning_rate=1e-4,    buffer_size=2_000,  # Started small, later increased    batch_size=256,    ent_coef="auto",    target_entropy=-2.0,    device=device,    tensorboard_log=log_dir,)

Stage-Based Training Approach:Rather than just running continuous training, I implemented a stage-based system that adjusted exploration over time:

class StageBasedTraining:    def __init__(self, model, vec_env, callback=None, start_steps=0, num_envs=2):        # Stage 1: High exploration (target_entropy = -2.0)        # Stage 2: Balanced phase (target_entropy = -3.0)         # Stage 3: Exploitation (target_entropy = -7.0, lower learning rate)

Custom Evaluation Callback:The most useful part was a custom callback that recorded videos during training:

class EvaluationVideoCallback(BaseCallback):    def _on_step(self) -> bool:        if self.eval_freq > 0 and self.n_calls % self.eval_freq == 0:            mean_reward, video_frames = self.evaluate_function(                self.model, num_episodes=self.num_episodes            )            # Save video and update best model if improved

This let me visually track learning progress and automatically save the best-performing models with videos.

The Stability Challenge and Practical Constraints

Even with dense rewards, SAC wasn’t always smooth sailing. I encountered several issues that taught me about the practical side of RL:

The Buffer Size Discovery: An Accidental Experiment

Looking back at my code, I discovered an interesting inconsistency that taught me about buffer sizes the hard way:

# Fresh training: very small bufferbuffer_size=2_000,  # Resumed training: much larger buffer  buffer_size=50_000,

This 25x difference was completely unintentional, but it created an accidental experiment. The small buffer meant:

Faster iteration during initial experimentation
Lower memory usage on my M3 MacBook (I was easily hitting memory limits)
Less stable training due to limited experience replay

The M3 Memory Reality:On macOS M3, I was constantly running into memory constraints. The 2K buffer was actually a practical necessity rather than a design choice—larger buffers would cause memory issues, especially with multiple vectorized environments and image observations.

Episode Length Evolution:I initially started with 300-step episodes, but later extended them to 700 steps. The reason: 300 steps were too short for me to complete teleoperation demonstrations. This change also affected my buffer size calculations—longer episodes meant each episode took more buffer space, so my effective “number of episodes in buffer” was different than I initially thought.

Platform Dependencies: The Colab Surprise

Stable-Baselines3’s vectorized environments were genuinely useful for throughput, but there was a huge platform dependency: On macOS with MuJoCo running on CPU, vectorized environments were surprisingly fast. On Google Colab, the same setup was painfully slow—likely due to weaker single-thread CPU performance and different MuJoCo builds.

Lesson: Always benchmark on your target platform. Performance characteristics can vary dramatically.

The Real Development Workflow: Storage, Videos, and AI-Guided Decisions

The Storage Reality:I was checkpointing aggressively—saving models, replay buffers, and VecNormalize stats every few thousand steps. Between the checkpoints and training videos, I ran out of my 1TB MacBook storage multiple times. The main culprit was the replay buffers at 3-4GB each—pretty typical for SAC with image observations. The other files were small in comparison:

Model checkpoints (~100MB each)
Replay buffers (3-4GB each - the real storage killer)
VecNormalize stats (small)
Evaluation videos (small)

When you’re saving every few thousand steps and keeping multiple checkpoints for safety, those 3-4GB replay buffers add up fast.

Videos as the Primary Feedback Loop:The evaluation videos turned out to be absolutely critical for understanding training progress. Unlike the reward curves, videos immediately showed me:

Whether the robot was learning useful behaviors
If it was getting stuck in local minima (like endlessly circling the cube)
When training was diverging before the metrics made it obvious
The quality of grasping and placement behaviors

Divergence Patterns in Video:Videos were especially revealing when the model diverged. You could see the policy just curling up in some specific pose regardless of what was happening—the robot would move to the same configuration every time, completely ignoring the cube position or any environmental state. This was much more obvious in video than in the reward curves, which might still show some variance.

Visualizing Entropy:The videos also worked as an excellent entropy visualizer. When entropy was too high, you’d see the robot making erratic, inconsistent movements—approaching the cube from random angles, jerky motions, inconsistent grasping attempts. When entropy was too low, the robot would be overly deterministic, potentially getting stuck in the same suboptimal strategy repeatedly.

AI-Assisted Training Decisions:I regularly copied my tensorboard training stats and shared them with Claude to get guidance on whether to interrupt training and resume from an earlier checkpoint. Since I didn’t know exactly what learning rate and entropy target to set for SAC, having an AI co-pilot helped me interpret the training curves and decide when to intervene.

This back-and-forth with Claude became part of my training workflow—especially when deciding whether unstable training was worth continuing or if I should revert to a more stable checkpoint.

The Resolution Debugging Story

This was my biggest “check your assumptions” moment. I had been training with 64×48 pixel observations for speed, and the results were disappointing. Policies would approach the cube but fail at precise manipulation.

I spent weeks tuning hyperparameters, reward functions, and network architectures. The rendered videos looked fine—I could clearly see the cube and bin. What was wrong?

The breakthrough came when I examined the actual tensor inputs to the model:

# Debug: save actual model inputsobs_tensor = env.observation_space.sample()plt.imshow(obs_tensor['pixels'].transpose(1, 2, 0))plt.title("What the model actually sees")plt.show()

The 64×48 images were severely degraded compared to my 480×640 rendered videos. Critical details for grasping—cube edges, gripper alignment, depth perception—were lost in the downsampling.

Switching to 640x480 input resolution immediately improved performance. The lesson: Always verify your actual model inputs, not just what you think you’re feeding it. Rendered videos can be misleading about what the policy actually observes.

Behavior Cloning: The Simulation vs. Reality Gap

I also experimented with imitation learning using manually collected demonstrations (implementation in scripts/train_bc.py). The behavior cloning itself worked well—policies could replicate demonstrated behaviors with reasonable fidelity. But collecting quality demonstrations revealed a surprising insight.

Using keyboard or gamepad controls, I struggled to actually complete the task successfully. I would succeed maybe 1 out of 5 attempts in simulation, spending most episodes fumbling with controls and failing to grasp or place the cube properly.

The insight that changed everything: Later, when I got access to a leader arm for real-world teleoperation, the difference was stark. Tasks I could barely complete in simulation became trivial in reality—I could consistently complete the manipulation sequence on nearly every attempt.

This isn’t about speed—it’s about task completion rate. The same human (me) went from a ~20% success rate in simulation to near-100% success rate with physical teleoperation.

The Double-Edged Sword:This creates an interesting tension in imitation learning. Struggling with simulation controls means you’re training on lower-quality demonstrations, but imperfect demonstrations might actually be better for learning robust policies—perfect demonstrations can be brittle and don’t show recovery behaviors.

My simulation demos included:

Recovery behaviors when grasps failed
Multiple approach angles when obvious paths didn’t work
Natural variation in timing and trajectories

Future Improvements: Hybrid Demonstration Strategies

Looking ahead, I could significantly improve the setup:

IK + Noise Demonstrations:Generate synthetic demonstrations using inverse kinematics with added noise for volume and consistency, following the ALOHA approach.

Leader Arm → Simulation Bridge:Connect a physical leader arm directly to simulation—combining intuitive 6-DOF control with simulation benefits.

Hybrid Dataset Strategy:

IK + noise for bulk demonstrations with good coverage
Human struggle for recovery behaviors and natural variation
Leader arm sim for high-quality expert demonstrations

Each source contributes different aspects of robust manipulation behavior.

VecNormalize: Mixed Feelings

Stable-Baselines3 offers VecNormalize to automatically normalize observations and rewards, but it couples environment statistics with your trained model. Deployment becomes more complex—you need to save normalization statistics alongside the policy.

My current preference: Handle normalization inside the policy network rather than as environment wrappers. It’s more explicit and avoids deployment surprises.

What I’d Do Differently

Looking back on these experiments, several lessons stand out:

Verify your actual model inputs early—don’t trust that preprocessing works as expected
Checkpoint often— training is slow, you don’t want to lose progress, and it’s sometimes helpful to revert back and change some hyperparameters (e.g. to avoid divergence)
Visualize everything—videos reveal problems hours before metrics do
Track your experiments and settings meticulously—there are settings scattered everywhere

The Settings Sprawl Problem:One of the biggest challenges was keeping track of all the different settings across multiple places:

Reward shaping: Defined in the task implementation
Camera resolution: Set in the environment creation
Environment setup: Goal vs. standard env, with or without wrappers
SAC training parameters: Learning rates, buffer sizes, entropy targets
VecNormalize settings: Observation normalization parameters

When experiments went wrong, it was often unclear which of these many settings was the culprit. Better experiment tracking from the start would have saved significant debugging time.

The Hyperparameter Uncertainty:Honestly, I didn’t know exactly what learning rates and entropy targets to set for SAC. Most of my hyperparameter choices came from a combination of:

Default values from Stable-Baselines3
Advice from Claude/ChatGPT
Trial and error based on video feedback
Academic papers (when I could find relevant ones)

This uncertainty made the video & checkpoint system even more valuable—I could experiment with different settings and revert when things went wrong.

Next up: How I integrated with LeRobot and what I learned about bridging different ML ecosystems in robotics.

Have you experienced similar challenges with RL stability or the simulation vs. real-world gap? What’s been your experience with different learning approaches?

Creating a Robotics Experimentation Environment: My Experience and Practical Lessons

Wed, 13 Aug 2025 00:00:00 +0000

Part 1 of a series on practical robotics experimentation

As part of my journey into robotics, I found myself facing a classic problem: I wanted to experiment with different learning approaches for robotic manipulation, but I needed a flexible playground where I could quickly test ideas without being locked into any single framework or workflow.

The result is gym-so100-c, a simulation environment built around the Standard Open Arm SO101 that bridges multiple machine learning libraries—Stable-Baselines3, the imitation library, and Hugging Face’s LeRobot—all in one cohesive environment.

The “Sim-First” Decision

Even though I had access to physical robots, I chose a sim-first approach for a simple reason: I’m more of a software person who enjoys the comfort of my home office and the flexibility to keep working while traveling, rather than spending long days in a lab.

This decision shaped everything about the project. I needed a simulation that was:

Physically realistic enough to eventually transfer to hardware
Fast enough for thousands of training episodes
Flexible enough to work with different learning paradigms
Simple enough that I could understand and modify every component

Why Build Another Gym Environment?

You might wonder: why not just use an existing simulation? I wanted something both flexible and reflecting my hardware platform well in simulation. The robotics world offers many compelling options.

The Simulation Landscape I Considered:

Isaac Sim was tempting—NVIDIA’s powerhouse with photorealistic rendering and advanced physics. But it requires a proper GPU setup, and I wanted to start experimenting immediately rather than waiting for hardware upgrades.
ManiSkill is an exciting newer option with great task diversity and modern ML integration. I’m actually quite excited to try this next—it seems to hit the sweet spot of realism and ease of use.
Gazebo/ROS represents the traditional robotics stack: mature, well-supported, with endless plugins. But the learning curve felt steep for someone coming from a pure ML background, and I wanted to focus on learning algorithms rather than robotics middleware.
PyBullet similar to MuJoCo. Also popular in RL enviornments.
Why I Chose MuJoCo + gym-aloha:The answer came down to immediate productivity. I could adapt gym-aloha and start experimenting within days, not weeks. It’s proven in the papers I was trying to replicate, has excellent contact modeling for manipulation, and enjoys a large ecosystem of compatible tools.
The Gym Interface Standard:OpenAI Gym (now Gymnasium) defines a standard interface that every reinforcement learning environment implements:

obs, info = env.reset()           # Start a new episodeobs, reward, terminated, truncated, info = env.step(action)  # Take an action

This simple interface is incredibly powerful because it means the same environment can work with:

RL libraries like Stable-Baselines3 (SAC, PPO, HER…)
Imitation learning libraries like imitation
Custom training loops or evaluation pipelines
Any future framework that follows the standard

The Evolution Plan:This environment is just the beginning. As I move toward more realistic scenarios, I’ll likely migrate to Isaac Sim or ManiSkill. But for rapid prototyping and algorithm comparison, this MuJoCo setup has been perfect.

Key insight: There are many good options. If your requirements are not very specific, look for something popular and that has something similar to what you need that you can quickly adapt.

This flows naturally from the question and sets up the technical details that follow.

Standing on the Shoulders of ALOHA

Rather than building from scratch, I adapted the gym-aloha project, which implements the dual-arm ALOHA platform used in several influential imitation learning papers.

The ALOHA Foundation:ALOHA (A Low-cost Open Hardware Arm) proved that effective manipulation learning was possible with relatively simple hardware. The gym-aloha implementation provided:

MuJoCo physics foundation
Well-designed observation and action spaces

My Adaptation:I modified gym-aloha for a single SO101 arm (5-DOF + gripper) to match my hardware target:

# Simple registration exampleregister(    id="gym_so100/SO100CubeToBin-v0",    entry_point="gym_so100.env:SO100Env",    max_episode_steps=700,    nondeterministic=True,    kwargs={"obs_type": "so100_pixels_agent_pos", "task": "so100_cube_to_bin"},)

Once registered, creating the environment is straightforward:

import gym_so100               # triggers env registrationimport gymnasium as gymenv = gym.make("gym_so100/SO100CubeToBin-v0")obs, info = env.reset()

The Task: Cube-to-Bin

I focused on one fundamental manipulation task: bin-a-cube. A red cube starts at a random position on the table, and the goal is to place it inside a fixed gray bin.

This task is deceptively simple but covers the core challenges of manipulation:

Perception: Locating the cube and understanding spatial relationships
Planning: Approaching the cube from a graspable angle
Control: Executing smooth, coordinated motion
Manipulation: Grasping, lifting, and precise placement

The MuJoCo scene includes:

Robot: SO101 single arm with position-controlled actuators
Workspace: Table, free-moving cube, and goal bin
Sensors: Joint positions, gripper state, and camera views
Sites: Tracking points for reward computation and success detection

Control Paradigms: Joint vs. End-Effector Space

Following gym-aloha’s design, I implemented joint-space control where actions directly specify target joint positions:

Aspect	Joint-space control
Action	Target joint positions → `data.ctrl`
Control loop	Actuators drive joints toward commanded positions
Learning	Policy learns in robot’s natural DOF
Transfer	Direct mapping to real hardware

I chose joint-space over end-effector control for cleaner transfer to my target hardware. While end-effector control (where you specify gripper poses and let MuJoCo’s constraints solve for joint angles) can be more intuitive, it adds complexity that I wanted to avoid initially.

What’s Inside the Environment

As of August 2025, gym-so100-c includes:

Simulation Environment & Tasks:

MuJoCo-based SO101 simulation environment
Cube-to-bin task with configurable reward shaping

Training Integration Scripts:

Reinforcement learning with Stable-Baselines3 (SAC)
Imitation learning with the imitation library
Training integration with LeRobot (ACT, Diffusion, VLAs)

Control & Data Collection:

Teleoperation via keyboard or gamepad
Episode recording for demonstration datasets
Dataset conversion to LeRobot format

Early Design Lessons

Physics Engine Choice:

MuJoCo was the obvious choice for its speed, stability, and excellent contact modeling. It works well on CPU, it was perfect on MacBook.

I am excited about IsaacSIM, but I don’t have a powerful GPU yet. I am open to trying other engines in the future.

Observation Space Design:

I experimented with different observation combinations:

pixels_agent_pos: Camera images + joint positions
agent_pos: Joint positions only (for faster training)
pixels: Camera images only (for vision-based policies)

The mixed approach (pixels_agent_pos) worked best, giving policies both rich visual information and precise proprioceptive feedback.This is also what ACT paper and most SOTA approaches are using.

Reward Engineering:

In Reinforce Learning an agent collects rewards interacting with the environment and tries to maximize them. The trick is to make such rewards that the agent does what you want from it and also that it can learn pretty well. Shaping the reward structure (designing the incentives) is called Reward Engineering and it’s suprisingly tricky to get right!

For example if you reward getting close to the block too much, the agent might chose to hover over it and never grab it. If you don’t give any rewards until the task is complete and if the task is too hard, there is no feedback to learn from.

I implemented both sparse rewards (success/failure only) and dense rewards with approach shaping. Dense rewards proved much more reliable for learning, though they required more careful tuning.

I also implememented HER (hindsight experience replay) that automates the reward shaping by trying to teach the robot behaviors from previous trajectories, but the results were mixed in my setup.

The Integration Challenge

The real value of this environment isn’t just the simulation — it’s the integration layer that lets me seamlessly move between different learning approaches. In the next post, I’ll dive into what I learned from training experiments with SAC, behavior cloning, and modern imitation learning methods.

But the foundation was crucial: having a single environment that could work with multiple learning paradigms, generate consistent datasets, and provide reliable evaluation metrics. Sometimes the unglamorous infrastructure work is what makes everything else possible.

Next up: Training experiments and what I learned about the practical differences between reinforcement learning and imitation learning approaches.

What’s your experience with simulation environments for robotics? Have you found certain design decisions that made experimentation much easier or harder? I’d love to hear about it.

My deep dive into robotics: Part 1

Thu, 24 Jul 2025 00:00:00 +0000

This post is the first in a series documenting my journey into robotics. I’m writing these to reflect on my progress and strategy, but also to help or inspire anyone considering a similar leap.

If you’re curious about AI, building physical systems, or carving out your own path into deep tech—this one’s for you.

Why robotics & my background

I grew up on a farm, so I’m no stranger to tedious manual labor. I also happen to be very lazy when it comes to physical work - as a kid I would never want to help out in the farm or doing chores. I’m not proud of it, but that’s true! I’d always rather be learning or building something than repeating the same task over and over.

Why not make robots do the boring work?

I’ve long believed that robotics—especially embodied intelligence—has huge potential to reshape critical parts of our world: infrastructure, food systems, care work. Europe, where I am based is facing an aging crisis and labor is getting more and more expensive and that drives the cost of housing, food and services. But seriously, I don’t think this trend needs to continue. Cheap electricity from atom/renewables + robotics can make labor cheap again and drastically improve quality of life for everyone if things go right (Yes, I’m a techno-optimist for sure!).

I’ve also been deeply into reinforcement learning for years. I remember doing the University of Alberta RL specialization on Coursera and being struck by the elegance of the math and the philosophical depth of the agent paradigm. It felt personal—figuring out how to learn, how to explore, how to act in an uncertain world. I could relate, as a human.

This project came at a natural inflection point. I had just finished a startup accelerator, moved back to Poland, gained access to a physical workshop with my brother, and met someone who introduced me to the Lerobot ecosystem.

During the startup accelerator, I had a lot of time to ponder what type of business I would like to do. Robotics was definitely something I was interested in, but I didn’t have a competitive edge, due to lack of practical experience.

I wanted to do something technically deep and fun—and potentially startup-worthy. Robotics fit perfectly.

Lerobot ecosystem really picked my interest and made actual robotics seem much more approachable and accessible. State of the art models on commodity, open hardware? Lead by Hugginface (a great brand name in the ML community)? Sign me up!

What types of robotics I wanted to learn

I’m focused on manipulation—robot arms, gripping, interacting with the world. I’m less interested in mobile robots for now, and more excited by the intersection of learning-based control and practical use cases.

My application areas of interest were agriculture and construction. Both are high-impact and under-automated. Both also demand robustness and real-world intelligence—qualities modern AI might finally deliver.

Timeline and the plan

I love learning and I’ve always loved to learn on my own. As an experienced self-learner, I’ve learned that investing upfront in a good learning plan pays off massively.

A few years back I read Ultralearning by Scott Young, and one key takeaway stuck with me: people often spend too little time researching how to learn best, picking the resources and making a plan. If you are willing to spend hundreds of hours learning, pick wisely.

I am not a robotics expert, so figuring out a curriculum all by myself would be hard. I also have a pretty strong science/engineering/software background, so I didn’t want to follow something generic. So, what to do?

Well, good that we’ve got powerful LLMs to personalize our learning paths.

I used ChatGPT Deep Research to help me generate a 12-week roadmap. Using Deep Research was important, because it causes LLM to ask clarifying questions and scour the internet for you, saving potentially tens of hours of research and analysis.

The input: I had a background in software, ML and RL, was aiming for deep hands-on knowledge, had ~30 hours per week to spare, and wanted to explore robotics manipulation with an eye toward AgTech or construction startups.

I already have some background in RL and ML. I want to learn about state of the art robotics to start a company. I don’t want to do a degree. I have time and money though. I am going to build and experiment with Lerobot from hugging face soon (waiting for last parts to arrive). I have access to 3d printers and heavy workshop. Create me a 3 month learning roadmap, with project ideas, assuming 30 hours of effort available per week.
I know python and I have many years of experience in software engineering.

See the full chat history for the plan creation here.

The output was ambitious: a full-stack robotics journey mixing ROS2, LeRobot, reinforcement learning, imitation learning, simulation, hardware, and even startup-aligned thinking. It was crazy - but in a fun, exciting way.

I put the plan into google doc and I have shown it to some people. They didn’t have major concerns apart from “woah, really?” - but I’m used to reactions like this.

See the plan as an exported Google Doc.

Why 12 weeks? This is part of my regular planning, I usually plan daily, weekly and quarterly. Twelve weeks is long enough to make real progress — but short enough to keep urgency high.

Plan vs Reality

I created my plan on 16 May 2025. Now we have 25th July - we are 10 weeks in. Realistically, I focused maybe 7 weeks on robotics between other projects and travels.

So how did it go?

I started out following the roadmap pretty seriously. For the first two weeks, I was all in—spending ~30 hours a week, doing two full Udemy courses on ROS2 and manipulation, and diving deep into kinematics and motion planning. I even assembled the LeRobot SO101 arm around that time.

But I never treated the plan as a rigid calendar. I thought of the “weeks” more like themed modules—open to remixing, repeating, or extending. I also gave myself permission to insert breaks between phases. It was aggressive by design, but flexible in execution.

Some things I deliberately skipped or reframed. For example, I didn’t do the vision pipeline in week 4—OpenCV felt outdated for where I saw the field going. I was more interested in learning-based approaches to perception. Similarly, while I didn’t use ROS2 with my SO101 arm, I did learn how to integrate ROS2 and Arduino in earlier work with a janky robot from the course. The skill was there—I just chose not to apply it in that form.

After two solid weeks, I reached a point where I felt I had enough foundational understanding to start building something real. That was the real pivot point: I asked myself, “What would it take to actually do a project?” That question reshaped my trajectory.

Instead of following the rest of the roadmap linearly, I shifted into execution mode. I was craving for a longer duration specific project - and I made another plan, this time a project focused one - Laundry Folding Robot inspired by aloha. See it here.

With my specific laundry folding robot in mind: I built a custom gym-style simulation matching the SO101 arm, and from there dove headfirst into reinforcement learning (mostly SAC), imitation learning (BC, GAIL, AIRL), diffusion policies, and VLA models like smolVLA and Pi0.

It didn’t match the tasks from the original plan — but at the same time, it was aligned with the curriculum I wanted to cover. The plan had 3 weeks fouced on RL, imitation and VLAs, I just did it from the point of a specific use case, instead of following the steps of the original plan.

Yup, and I managed to teach the robot how to throw the bin to the box.

(even though some of my teleoperation demonstrations were pretty poor - see here - it’s not easy to control a robot!)

Reviewing the original plan: I didn’t touch sensing, SLAM, or domain-specific deployment yet. I already had some experiences with sensing (courses in the past, etc), but it’s definitely an area to dive deeper into, especially when I get into mobile robots.

The remaining: “Week 7: Advanced Sensing and Actuation – Toward Field Deployment”, “Week 9: Sim-to-Real Transfer and Scaling Up”, “Week 10: Domain Focus – Agricultural and Construction Robotics Deep Dive” still seem very useful and interesting.

I did some data collections in real life, but my setup was super crappy:

(picture from before I invested in a tripod and more cameras)

The last week of the plan was meant to be spent on reflection and I’m doing it ahead of schedule since the execution has gone off plan, while still on track for the high level goals.

Talking to people

My official learning plan didn’t include speaking with others. But I’ve come to appreciate just how much easier — and more energizing — hard things become with the right people around.

I visited two robotics labs in Warsaw and spoke with researchers there. One lab focused on swarm robotics and was led by someone with a clear entrepreneurial mindset. It was inspiring to see someone thinking about scalable systems and real-world impact. The other lab worked on human-robot interaction, especially with children — a domain I hadn’t seriously considered before, but one that sparked new curiosity.

Both labs invited me to collaborate and offered access to their robots. That kind of openness was encouraging. While neither was working directly with deep learning for robotics — the area I’m most drawn to — these conversations broadened my view of what robotics can be. They gave me strength and a sense of possibility. At one of the labs, I even reconnected with an old friend I hadn’t seen in over a decade. It was unexpectedly meaningful to find someone I knew already immersed in this space.

I also reached out to people at Warsaw-based robotics companies like nomagic.ai. So far, those connections haven’t led anywhere, but I’ll keep exploring. I missed the LeRobot worldwide hackathon, but joined the Discord and have been in occasional contact with the friend who introduced me to the platform.

I started this journey solo — with a roadmap, tools, and focus. But if I do pursue something deeptech, I know I’ll want collaborators. These early conversations reminded me that the right people can shift what feels possible.

Reflection on the learning

The roadmap was wildly ambitious. Some topics were PhD-level, and there’s no way to do justice to them all in three months. Having a tight schedule was helpful to prevent me from going to deep on any specific topic, it would be fun, but I didn’t want to lose the sight of the bigger picture.

Having a plan really helped, even if I didn’t stick to it strictly. It helped me to have a somewhat structured personalized learning plan.

The biggest takeaway: mixing structured learning with project-driven deep dives is incredibly effective. Courses gave me confidence; projects gave me momentum.

Some parts of the roadmap (like OpenCV pipelines or QA protocols) felt less relevant with how fast deep learning is progressing—but that’s fine. It’s all about strategic depth, not completeness.

Reflection on the plan

The 12-week structure gave me an early push, but I was always open to adjusting it. I didn’t treat the roadmap as a rigid calendar—it was more like a menu.

In retrospect, I think mixing theory and practice is hard to pre-plan. That’s where I deviated the most. But having the map made it easier to navigate.

What else from the plan do I plan to do?

I still want to:

Build a basic vision pipeline + scripted action demo.
Explore field deployment topics (SLAM, navigation).
Learn more about sim-to-real strategies.
Do a more integrated prototype + test cycle for a specific use case (e.g. fold my laundry!)
Revisit AgTech and construction applications with more realism. Or pivot to another industry entirely (e.g. care sector).

Additionally I plan to:

Make better demonstrations
More good demos with real robots
Build more robots - LeKiwi and Xlerobot are next!
Instead of e2e VLAs also try out LLM + IK over MCP
Invest in better simulation - either IsaacSIM or Maniskill

Using AI for learning / projects

I used AI extensively during this learning journey:

it helped me crafting my plan and find the right resources to learn from
explaining specific concepts
drafting anki cards so I can remember the concepts better
help troubleshooting installations
help coding and ML training
emotional support during RL training (seriously, it was needed, many models are prone to diverging) together with hyperparameter tuning

I used chatGPT, Claude, Gemini and VSCode with Copilot (powered mostly Claude Sonnet). I briefly tried gemini CLI, but this tool has much more to offer. I just scratched the surface.

I created a Project within chatGPT where I included my plans as project files and that helped a lot to focus the AI.

My project instructions:

You are a robotics/RL expert and an expert educator. You are my learning assistant. Think more coach than a teacher.
Be concise and do not come up with suggestions without understanding what is needed from you. Ask a lot of questions before suggesting anything or sharing an opinion. I want to think independently, help me with that. Any time you offer a suggestion, make sure to justify why it’s needed and why it’s better than alternative approaches.

Lerobot - open hardware, tutorials, easy to access models and datasets - maybe a bit confusing and unstable at times, but amazing! Additionally they have a lively community!
Reinforcement Learning Course - very approachable
CS 285 Deep RL Course - not for the faint of heart, lots of SOTA techniques
Modern Robotics Course - I would call it classic robotics in the context of my learning, but it covers a lot of useful fundamentals
ROS2 Basics Course - was useful for understanding ROS2, URDF (how robots are described in software), Gazebo for simulation
ROS2 Manipulators Course

Conclusions & tips

If you’re aiming for a deep-tech startup, playing the long game on learning is worth it.

You need a mix of practical experience and theory to back it up
Structured courses are the best way to acquire theoretical knowledge
Practical projects are the best to keep motivated while learning a plethora of tools and approaches
Plans are tools, not rules. Break them if you need to.
Having a clear project goal helped me stay focused and curious.
Being overwhelmed is normal. Keep going anyway. Use friends and Claude for emotional support.
Buying in literally by spending money on hardware made it much more real

Next up: in Part 2, I’ll dive into my current setup, I will share my simulation env and scripts I used to integrate with:

mujoco / gym
stablebaselines3 (RL)
imitation (IL)
lerobot (Diffusion, ACT, VLAs)

including training, teleoperation, storing/uploading datasets in multiple formats and eval. I’m happy to share my code in help that it helps people with integrations as these tools can be overwhelming.

Generating Hundreds of Consistent Illustrations with Gemini Image Generation

Sun, 20 Jul 2025 00:00:00 +0000

A deep dive into building an automated illustration pipeline for storytelling applications

Introduction

AI image generation has revolutionized creative workflows, but there’s a significant difference between generating a single stunning image and producing hundreds of consistent illustrations for a complete project. When building storylearner.app, we faced the challenge of generating book illustrations that maintained visual consistency while telling compelling stories through imagery.

We’ve used powerful Gemini multimodal models, mostly because they are fast and because the experimental ones are available for free.

This article explores the technical and creative challenges of large-scale AI illustration generation, showcasing techniques for achieving visual consistency and building robust pipelines that can handle the complexity of full book illustration projects.

Below are images created for a chapter of a book adapted for the storylearner platform:

The article has a companion colab notebook, so you can play with the examples yourself.

The Consistency Challenge

Understanding Visual Consistency in AI

AI image generation models operate somewhat like a company of talented artists, each with amnesia. Every generation is essentially a fresh start unless you provide explicit context. This creates unique challenges when you need to maintain character consistency, style coherence, and narrative flow across hundreds of images.

Consider this simple experiment: generating three images of “the same pig with wings and a top hat flying over a futuristic city” in different weather conditions. Even with identical prompts and seeds, subtle inconsistencies emerge—eye colors change, proportions shift, and the overall character can feel different.

The Impact of Seeds and Prompts

Seeds act like selecting a specific artist from your AI company. Using the same seed with identical prompts yields consistent results, but even small prompt variations can dramatically alter the output. We discovered that:

Same model + same prompt + same seed = identical results
Same model + same prompt + different seed = completely different character
Same model + slightly altered prompt + same seed = often produces a different character entirely

This sensitivity means that scaling up requires careful orchestration of all these variables.

Building Consistency Through Reference Images

The Reference Image Approach

The most reliable method we found for maintaining character consistency involves using reference images. Here’s how it works:

Generate an initial character/scene using carefully crafted prompts
Upload this image as a reference for subsequent generations
Include explicit instructions like “Use the supplied image as a reference for how the pig should look like”

This approach significantly improves consistency, though it can also cause new issues in some very specific cases.

With the following reference image:

You can get this different, but consistent one!

Here is an example of triggering a specific case:

This specific case can happen when a prompt is pretty similar to the one that generated the initial image and the seed is identical.So, when using the reference image and similar prompt, I would actually recommend to change the seed or not set it.

See the reference colab for details.

Practical Implementation

# Upload reference imagefiles = [client.files.upload(file="reference_character.png")]# Create parts with reference and promptparts = [    types.Part.from_uri(        file_uri=files[0].uri,        mime_type=files[0].mime_type,    ),    types.Part.from_text(        text=prompt + "\nUse the supplied image as a reference for character appearance"    ),]# Generate with referenceresponse = client.models.generate_content(    model=IMAGE_MODEL,    contents=parts,    config=types.GenerateContentConfig(        response_modalities=['Text', 'Image'],        # No seed set.    ))

The Storylearner.app Illustration Pipeline

High-Level Architecture

Our production pipeline consists of three main stages:

Idea Generation: Story text + guidelines → 3 illustration concepts per scene
Idea Selection: Multiple concepts → best ideas chosen for the complete set
Image Generation: Selected ideas + style guidelines + reference images → final illustrations

Stage 1: Brainstorming Illustration Ideas

Rather than feeding story text directly to image generation (which often produces poor results), we separate conceptualization from execution:

class IdeaGenerator:    def generate_illustration_ideas(self, text: str, context: str):        prompt = """        You are a visual scene designer. Based on the story below,         describe 3 different highly detailed and imaginative illustration ideas.         Do not include any people or humanoid figures. Focus on setting,         atmosphere, lighting, symbolic objects, and environmental storytelling.                Story: {text}        Context: {context}        """                # Returns 3 detailed scene descriptions per text excerpt

This approach generates rich, detailed scene descriptions that serve as blueprints for image generation.

Stage 2: Intelligent Selection

To avoid repetitive illustrations (like “a ship, a ship, a ship” in a sea voyage story), we use an AI selector to choose the best combination of ideas:

class IllustrationSelector:    def select_illustrations(self, chapter):        prompt = """        Choose the best idea for each illustration considering that:        - The set should be diverse        - Illustrations shouldn't contain people        - Prefer illustrations matching the provided titles        """        # Returns optimal selection indices

Stage 3: Consistent Generation

The final generation stage uses:

Style guidelines (detailed visual specifications)
Reference images for style consistency
Persistent chat sessions for maintaining context
Retry mechanisms for handling API limitations

Visual Guidelines and Style Consistency

Crafting Effective Style Guidelines

We developed comprehensive style guidelines that go beyond simple style names:

Style: watercolorTechnique: Combine soft watercolor washes with fine ink line work for contrast and detail.Brushwork: Embrace visible brush strokes, blooming, and natural texture.Ink Lines: Use varied line weights for depth; apply cross-hatching or stippling for texture.Color Palette: Limit to a few harmonious hues with gentle gradations.Forms: Use simplified, geometric shapes; focus on essence over detail.White Space: Treat negative space as part of the composition.Texture: Highlight watercolor paper's natural texture and color variation.Atmosphere: Create light, airy scenes with openness and subtle contrast.Aesthetic: Preserve a hand-drawn look—embrace imperfections and human touch.

Chat-Based Generation for Context Continuity

Using persistent chat sessions helps maintain consistency within illustration sets:

def generate_set_of_illustrations(ideas_with_file_paths, pass_image=True):    chat = client.chats.create(        model=IMAGE_MODEL,        config=types.GenerateContentConfig(response_modalities=["Text", "Image"]),    )        # Initialize with style guidelines and reference image    initial_prompt = f"""    You are a creative artist helping on an illustration project.    Create {len(ideas_with_file_paths)} beautiful illustrations.        VISUAL GUIDELINES:    {VISUAL_GUIDELINES}    """        # Generate each illustration within the same chat context    for idea in ideas_with_file_paths:        response = chat.send_message(format_illustration_prompt(idea))        # Process and save generated image

Avoiding Common Pitfalls

Critical Design Decisions

Through extensive experimentation, we identified several key strategies:

Avoid Human Close-ups: Character face consistency is extremely challenging. Focus on environmental storytelling instead.

No Violence or Gore: Keep illustrations family-friendly and avoid content that might trigger safety filters.

Diversify Scene Types: The selection stage prevents repetitive imagery across the complete set.

Decouple Ideation from Generation: Separating concept creation from image generation improves both quality and debuggability.

Real-World Example: Illustrating The Three Musketeers

Let’s walk through illustrating a chapter from The Three Musketeers:

Context and Settings

First, we establish the story context:

Setting: France, primarily Meung and Paris, early 17th centuryHistorical Context: Political tensions between French monarchy and Cardinal RichelieuMain Characters: D'Artagnan, Athos, Porthos, Aramis, Cardinal Richelieu, Milady de Winter

Generated Ideas

For the chapter opening, our system generated these concepts:

“The Jolly Miller Inn Chaos”: Exterior scene with a yellow pony, scattered debris, and dramatic lighting hinting at recent altercation
“Broken Sword”: Close-up of shattered steel on cobblestones, symbolizing lost honor and broken dreams
“Inn Kitchen Aftermath”: Dimly lit interior with earthenware, bandages, and flickering candlelight

Selection and Generation

The selector chose the most diverse and narratively appropriate ideas, which were then generated using our reference image and style guidelines, producing illustrations that maintain visual consistency while telling the story effectively.

Key Insights and Best Practices

Consistency vs. Perfection: Perfect consistency isn’t always necessary—visual coherence in style and mood often matters more than exact character matching.
The Artist Analogy: Think of AI models as artists with amnesia. You need to provide context, references, and clear instructions for each interaction.
Pipeline Modularization: Breaking the process into idea generation, selection, and execution improves quality and maintainability.
Style Guidelines Matter: Detailed, specific style descriptions work better than simple style names.
Reference Images Are Crucial: Upload and reference style examples for best consistency results.
Long Sessions are Fragile: It often works until a point, and at some point it fails poorly, e.g. inserting objects from a previous illustration into the following ones.

Conclusion

There is still a huge gap between a carefully handcrafted demo on an AI company blog and practical usage of the technology at scale. Things don’t work out well straight out of the box, but you can make these amazing tools work for you with help of systematic thinking about consistency, quality control, and robust engineering practices.

While challenges remain, I hope you’ll enjoy the techniques we’ve developed at storylearner.app to power your own projects!

Escaping LLM piping mess with nifty engineering

Wed, 25 Jun 2025 00:00:00 +0000

In this post I’ll walk through how I upgraded a set of tangled Python notebooks—responsible for thousands of LLM calls—into a robust content-adaptation studio powered by:

an async FastAPI pipeline,
a disk-first Next.js frontend, and
a small suite of custom CLI tools.

Re-engineering the stack was essential for my own sanity: the notebooks were fragile, slow to iterate on, and far too labor-intensive to babysit.

I was also facing content quality challenges that were pretty much impossible to address in the old code base, that re-engineering unlocked.

My hope is that the story also nudges you to build (or level-up) your own tooling instead of settling for one-off notebooks.

We’ll cover:

Engineering constraints – huge text volumes, strict meaning preservation, multiple target languages.
The original notebook setup – what worked and where it hurt.
Pain points – why small hacks no longer cut it.
The new architecture – key design choices, novel elements (with screenshots), and how they solve the earlier pain.
Outcomes & takeaways – higher quality, less toil, faster experiments, and patterns you can reuse in your own LLM workflows.

Shape of the problem

Shape of the problem:

large quantities of text that need to be processed in a very specific way:
- retain meaning of the original (no text disappearing or altered significantly)
- consistent between parts
- translated & simplified
resulting content needs to consistently have good quality
- automatic quality assessment and quality repair (revisions)

A similar challenge would be relevant in translating legal documents, healthcare documents, etc.

Story Learner Book Adaptation needs

In my project StoryLearner I offer adapted books for language learning at a specific level. For example, “Las Aventuras de Sherlock Holmes, in A2, Spanish”.

We use LLMs for both language/level adaptation as well as for illustrations.It’s a lot of LLM calls (easily thousands for a single book), because books are long and there are many elements for a single adaptation.

A book has chapters, chapters have parts for easier reading. Each book/chapter/page has an custom illustration. Additionally, there are titles and descriptions to be adapted and transcribed.

Books are long and we want the adapted text to retain the meaning, while making the language simple (aligned with the target level), natural sounding and correct.

Trouble with a flaky, slow pipeline and hard to assess output

The pipeline was was quite a feat! It was a lot of LLM calls, built mostly in colab/jupyter notebooks.

RAW BOOK   |   v[book_stripping]   |   v[chapter_extraction]   |   v[chapter_simplification] (English)   |   v[chapter_partification] (English)   |                              \   |                               \    |                                ---> [illustration_generation]   |   v[adapt] ──▶ [Lang 1]   │        [Lang 2]   │        [Lang 3]   │        [Lang 4]   │        [Lang 5]   │        [Lang 6]   |

Illustration generation was on its own a pretty interesting pipeline (more about it in a separate post!).

Here is one of the adaptation notebooks:

As you can see, it has its own table of contents on the side. It’s easily thousands of lines of code and prompts. And the hundreds of outputs (text and images) could make it very, very, long. To the point that it would have rendering issues.

On top of it there was:

Separate notebook for book_narration
Another notebook for upload to storylearner (via API).

The “pipeline” worked. It serialized partial outputs in a way that partial redos/continuations were possible, it offered decent visualization. It was adjustable (just add/tweak a notebook cell!).

However it was fragile and assessing quality/redoing content was painful and slow. And it was very frustrating to me, especially since the quality was important to my partners (language schools).

Main pain points

LLM reliability issues (no resources, surprise safety controls kicking in, running out of quota or LLMs not following instructions) broke downstream steps.
The adaptation process was slow, it wasn’t taking advantage of paralellization well
The pipeline was already so brittle enough that meaningful experiments were nearly impossible
Colab/notebooks encouraged slapping things together instead of proper engineering with encapsulation and tests
Python notebooks having rendering bugs because they were so long and had so many outputs (large text/many images)
Low confidence in language level, name/format consistency, and preserved meaning without having a strict review/repair process and having some examples of problems with quality.
Reviewing 60+ chapters across six languages was slow and manual - was infeasible for me.
A copy of an adaptation notebook per book (for visualization/auditability) was duplicating the code and making it harder to maintain

New Content Adaptation Tools

I haven’t built everything at once. I started with a frontend using the existing disk format of book adaptations, then as it was easier to see what was going on, I progressively built more and more backend migrating specific functionalities. Fast API gives a nice UI out of the bat to call APIs, which was nice for trying things out. But the workflows were simply to long to drive them by hand, so that is how CLI tools came to be.

CLIs/Frontend were in big part written by copilot coding agent. I also extensively discussed the component prompts with LLMs 😀.

Backend – bookadaptation (FastAPI)
- all functionality behind endpoints, all async / internally parallel where safe (28 separate endpoints)
- All IO and LLM calls done async
- Pipeline stages are classes; 11 adapter subclasses (Gemini Flash, Gemini Pro, GPT-4o, etc.).
- skip and use_cached flags run a no-op or reuse artefacts while the file tree stays unchanged.
- STAGE_DEPENDENCIES mapping declares primary & secondary inputs and outputs for every stage.
- Hierarchical on-disk structure; files get a _{revision_number} suffix for multi-round outputs.
- Heavy use of controlled generation (using schemas)
- SQL instrumentation: All prompts, schemas, settings, outputs, latency, and errors logged to SQLite.
Frontend – ContentTools (Next.js Server Components)
- Specialised views:
  - Book language adaptations
    - Illustrate all artefacts of the adaptation pipeline (inputs, outpus)
    - Easy debugging of what happened during the QA
    - Easy comparisons between experimental implementations
    - “Final for publish” view.
  - Illustrations:
    - All book illustrations as grids (chapters, chapter parts)
    - Illustration deep dives - ideas and the best ideas
- Reads JSON & WebP directly from disk—no extra HTTP hop.
CLI tools (all async friendly python)
- adaptation CLI
  - Adapts and revises every chapter until it’s good enough
  - Drives the pipeline stages implemented in the service (many endpoints right)
  - Gracefully retries
  - LLM driven decisions: Uses output from the overall review to finish the chapter or go for more rounds or partially skip stages
- narration CLI
- illustration CLI
- publishing CLI

Everything runs locally, but could as well run on a server.

Yes, I ended up building a lightweight custom pipeline orchestration… 💀

Flow of a book adaptation with automatic QA (multiple revisions)

RAW BOOK   |   v[book_stripping]   |   v[chapter_extraction]   |   v[chapter_simplification] (English)   |   v[chapter_partification] (English)   |                              \   |                               \    |                                ---> [illustration_generation]   |   v[adapt] ──▶ [Lang 1]   │        [Lang 2]   │        [Lang 3]   │        [Lang 4]   │        [Lang 5]   │        [Lang 6]   |   v[review_chapter]  ←──────────────┐   |                             │   v                             │[revise_chapter]                │   |                             │   v                             │[review_consistency]            │   |                             │   v                             │[revise_consistency]            │   |                             │   v                             │[review_meaning_cohesion]       │   |                             │   v                             │[revise_meaning_cohesion]       │   |                             │   v                             │[review_titles]                 │   |                             │   v                             │[revise_titles]                 │   |                             │   v                             │[review_chapter_title]          │   |                             │   v                             │[revise_chapter_title]          │   |                             │   v                             │[review_overall]  ──────────────┘   |   v                             [promote_content]   |   |   +--> [book_narration] (from adapted text)   |   +--> [upload / publish]

QA rounds are controlled by the output of the review_overall stage.

Examples of frontend enabling fast QA

Example of problematic images that can be ’easily spotted’ by a trained eye.

Chapter level issues overview for an adaptation workflow:

Part level debugging/review of what was suggested/applied in the QA pipeline:

Review and Revision—kept deliberately apart

One key design choice was to decouple “reviewer” from “reviser”.

The reviewer node reads the necessary context and produces a structured list of issues,while the reviser node sees only the affected slice plus those suggestions.

Why keep them separate?

Audit clarity
The reviewer’s JSON lives as its own artefact, so you can diff, grep, or hand-editthe feedback without touching the text itself. It’s also easier to audit automatic revisions and spot ‘additional helpfulness’.
Smaller prompts, cheaper calls
A reviser that operates on just the target part + suggestions uses far fewer tokensthan one that re-ingests the whole chapter.
Less collateral damage
Narrow context means the reviser can’t “helpfully” rewrite good paragraphs in other sections or even just completely forget them.
True parallelism
Parts are context-isolated, so multiple revisions can run concurrently—no giant chapter-wide lock.
Targeted rollbacks
If a revision introduces a new issue, it’s easy to rollback.

What the flow looks like

review_meaning_cohesion scans consistency_revised_stories
and writes meaning_cohesion_review_1.json:

 { "issues": [     {{         "part_number": 1,         "suggestion": "Change sentence '...sentence...' to '...corrected sentence...' to ensure consistency with previous parts.",         "reason": "Inconsistent character name across parts.",         "severity": "high"     }},     ... ], }

During the revise_meaning_cohesion stage, revisions to specific parts are applied concurrently, e.g. reviser for part 8 only sees the text of part 8 and the suggestions for part 8.

Other novel elements

Stable DAG via no-op nodes – skipping or caching never changes filenames or dependencies, so UI and controller logic stay simple.
Declarative STAGE_DEPENDENCIES – each endpoint validates its own inputs and fails fast if artefacts are missing.
Pluggable adapter subclasses – swapping models or prompt strategies is a config change, not a refactor.
Direct-disk reading Server Side React Components – Suprisingly trivial frontend code.
SQLite error forensics – a single query surfaces “prohibited-content” or other LLM failures.
Hot-reload mid-run – tweak prompts or error handling while a 60-chapter fan-out is running; retries pick up the change without restart.
WebP illustration storage – generated art compresses very well; files are roughly 10× smaller than raw outputs.
LLM-driven controller decisions – the CLI uses the review_overall output to decide whether to launch another revision round and which stages to skip.

Personal wins

The biggest win is defending my personal sanity.

I no longer have to babysit a set of fragile notebook based pipelines based on fallible and untrustworthy LLMs.

Higher confidence in quality – every chapter passes level, consistency, meaning, and overall reviews—automatic multi-round fixes if needed.
Far less toil – no more babysitting fragile notebooks; the pipeline self-checks and fails early.
Fast, parallel experimentation – new adapters or prompts run side-by-side with hot-reload; iteration is “super fast and fun.”
Cost flexibility – total spend is higher (as there are more LLM calls), but the modular design lets me fall back to cheaper or local models whenever I choose.

And the cool thing is that building this tooling was heavily accelerated by a coding assistant/agent, so it was significantly faster and more fun than I would have expected from the scope of the reengineering.

Why I Pivoted StoryLearner Instead of Giving Up

Sun, 18 May 2025 00:00:00 +0000

I have a secret for you: the first version of storylearner.app looked pretty different!

It had a pretty compelling story generation feature, where you could influence the story, pick the characters, and so on…

It looked good — the illustrations were very compelling, and the stories were pretty fun. But I ripped it away. In this post, I’ll dive into how I realized I needed to make the change, what to change, and how I executed it.

The First Reality Check and Discouragement

In the beginning, I was riding the creative high of building something new. I had done a decent amount of research, and the space didn’t look too crowded. I felt motivated and inspired.

But over time, as I started talking to people and working on the product, the landscape began to shift.

I discovered tools like Language Reactor and started getting aggressively targeted by competitor ads — apps I hadn’t even known existed before.

It was like pulling at a thread: the more I looked, the more players I uncovered, especially once I started working on the mobile alpha. That’s when it hit me — building something around stories wasn’t as novel as I’d thought, and the market was crowded.

That realization was discouraging, but not all at once.

I’d alternate between moments of “oh no” and renewed optimism: yes, the market is competitive, but I have a unique angle, right? That sustained me for a while.

But the final blow came when I uncovered a wave of mobile apps doing very similar things (like Redle - Rapidly Learn Languages with News & Stories). It was harder than ever to believe I had a differentiated product.

I also became increasingly skeptical about the viability of a subscription model in such a saturated space — people are tired of subscriptions. I started seriously considering quitting.

Thought Process Behind the Pivot

Selling individual books had crossed my mind before, but combining subscriptions and per-product purchases seemed too complex for the initial version of the product.

When the discouraging signs piled up, I took a step back. I wasn’t afraid to walk away — I’d had fun, I’d learned a lot, I’d shipped something. But I also had a nagging feeling that there was still something worth salvaging.

At that point, I had standalone stories, story packs, and a working prototype of a book adaptation pipeline. That pipeline was actually a pain to build — adapting books well is hard. You need to simplify language to the right CEFR level, maintain good translations, generate quality illustrations, and chunk everything into manageable parts. But that difficulty made it valuable.

Books are inherently engaging, and the idea of breaking them into short, predictable chapters — inspired by both Duolingo and Brandon Sanderson’s writing style (I’m a huge fan!) — made them even more learner-friendly.

So I made a call: drop the generative story feature, focus fully on adapted books, and build the product around that. It felt like a tighter, more focused bet.

Books are easier to understand and sell: people already look for language books. They can try a few chapters and decide whether they want to pay — no long-term commitment. That’s easier for users, even if it’s potentially worse for recurring revenue. But let’s be real — if no one subscribes, there’s no revenue anyway.

Why the New Approach Is Better and More Promising

The new direction is more aligned with how people actually think. When someone hears, “You can read Sherlock Holmes in simplified Spanish, with sentence-by-sentence narration and illustrations, plus word lookups and review tools,” they get it. That’s an easy pitch.

Compare that to: “You can generate custom AI stories with predefined characters in various CEFR levels…” It’s more abstract, the stories seem arbitrary. It sounds like yet another language learning gimmick, a thing that you probably don’t need.

Generative stories were fun and they looked good — hiding that feature wasn’t easy. But in practice, the value of curated, thoughtful content is higher.

It has more practical engineering benefits too. It’s reusable, more stable, and cheaper to generate at scale. With LLMs being flaky in both quality and uptime, having static, high-quality content is a strategic advantage.

Books also resonate emotionally. There’s a sense of identity and aspiration tied to finishing a real book — even a simplified one. And a known title gives people a reason to care.

Additionally, there was a pretty big difference in positioning that unlocked a new distribution channel. A learning app competes with the traditional language school, but a book? A book would be complementary, it doesn’t threaten the traditional teaching model. It adds an additional way to monetize learning to people who are already paying for learning. One important lesson I learned in the past is that selling to people who don’t want to spend money can be very frustrating, it’s much easier to sell to people who are already spending on something similar.

Plan and Execution of the Pivot

I planned the pivot in three parallel tracks: content, app changes, and marketing.

App changes, surprisingly, were faster than I expected. I was productive and used Copilot and Claude to speed things up. I restructured the app to make books first-class entities. I added audio narration support, Stripe integration for purchases, a chapter-based reader experience, and a reworked UX with a catalog and library view.

Then, I adapted three full books. That part turned out to be trickier than expected. LLM outputs needed QA and lots of checkpointing, sentence splitting caused bugs in TTS generation, and with 800MB per book (mostly images), even uploads became a challenge - one of the books had more than 700 hundreds of illustrations, and I also generated per sentence narration files. I had to parallelize file uploads and add more automated checks to make the pipeline robust.

Marketing and launch prep are still in progress. I created a new landing page and started working on blog content, but the real push is still ahead. I am also talking to people in real world and the response seems to be enthusiastic.

Reflection

Looking back, I’m glad I pivoted and didn’t give up.

Storylearner is a small, but real product — focused, scoped, and clear in what it offers. It won’t be a unicorn, but it doesn’t have to be. I’m still proud of it. The build is done, but can be extended and improved. Costs are low. Risk is capped.

The next phase is about getting it in front of people. Success will depend on distribution and my marketing skills. But at least now I’m selling something people intuitively understand. It also doesn’t compete with language schools — it complements them — which opens up potential partnerships and new distribution channels.

I’m cautiously optimistic. I didn’t quit. I refocused. I’m not tied down, and I still have upside. Wish me luck!

How I ended up building storylearner.app

Sat, 17 May 2025 00:00:00 +0000

I’ve always been drawn to self-directed learning. My interests have been pretty diverse - I would spend countless hours on math, science, soft-skills, movement skills (juggling, handstands) as well as languages.

A long standing language learning obsession

Over the years (starting as a teenager), I’ve seriously studied eight languages — not because I had to, but because I found it fun, challenging, and fascinating. Through trial and error, I’ve learned a lot about what actually works for adult learners learning on their own. And I also learned about all possible pitfalls, the main of them being, abandoning the project and losing all the progress due to forgetting.

But I’ve been successful in learning Spanish.

Reading, in particular, became my core of my learning. When I was learning Spanish, what helped most wasn’t drills or apps — it was regular reading. I started with graded readers and eventually moved on to novels. That reading habit didn’t just build vocabulary — it kept the language alive.

For most learners, jumping straight into native content is simply too hard. Instead, you can reach for graded readers - books adapted for your level.

But my experience with traditional graded readers wasn’t great. The selection was limited, the quality was often underwhelming, and there was no audio or pronunciation support. Later, I found other tools that approached reading through comprehensible input, but I wasn’t aware of them at the time. And even now, I think there’s still a gap: reading remains one of the most powerful ways to acquire a language — but if you aren’t fluent yet, it’s rough out there.

That gap stuck with me. Especially as I started learning Portuguese about two years ago — which, unlike Spanish, presented real pronunciation challenges for me. I had already experimented with TTS in a side hustle and taken several deep learning courses, so I started thinking: what could I build to actually help?

Struggle to find something that deeply resonates

At the same time, I was in a Founders Startup Accelerator, exploring B2B ideas and working with different cofounders. I kept chasing what the program was designed to produce — a validated, fundable B2B startup.

It was intense - lots of brainstorming, early excitement, and the slow crush of reality when things proved harder than expected. I struggled to get a personal fit with a validated business problem. After parting ways with my second cofounder - who was amazing, but very focused on the industry that I didn’t care deeply about, I decided to stop forcing it and just build something that genuinely excited me.

Leaning into my genuine excitement

So I started experimenting with a story-based reading tool. I wasn’t committing to it — just exploring something that felt fun and energizing.

Then a funny thing happened: I had planned a two-week paragliding trip in the south of Spain… (with the intention to take a step back from B2B SaaS and catching sunshine) and it rained the whole time. Paragliding was cancelled, so I rerouted to Málaga, found a coworking space called The Living Room, and settled in. I ended up spending two very productive weeks there — surrounded by friendly nomads, good energy, and people who were genuinely curious about the project. In that time, I made massive progress and storylearner started to take shape.

Later, I paused the project again to try one last cofounder collaboration. But it didn’t lead anywhere, and by the time the investment committee pitch came around, it was too late to get traction on anything new. So I returned to storylearner — not as a fallback, but because it was the one thing I wanted to keep building.

What started as a light experiment quickly turned into weeks of deep development. I built an AI-powered content pipeline to adapt public domain books into simplified, bite-sized chapters with sentence-level audio and illustrations. Along the way, I pivoted from generic story generation to adapted books — a format that’s structured, engaging, and rich in educational value.

Why it fits?

I’ve spent most of my career working in backend and infrastructure. I enjoy serious, deep engineering, but at the same time I really like building customer facing products. It’s an itch that I like to scratch. I’ve built up full-stack skills through side projects.

Storylearner is something I can build solo, keep costs low, and ship quickly - it gives me space to live my life and explore other ideas as well. But more importantly, it brings together the things I actually care about — language learning, AI, reading, and creating tools that support growth.

I also believe there’s a real opportunity here. GenAI is incredibly powerful, but it still involves a lot of DIY friction. Meanwhile, tools like Duolingo succeed because they make learning smooth, convenient, and even addictive. I think reading deserves the same treatment — and I want to help make it easier, more attractive, and more rewarding for language learners. Especially by unlocking the goldmine of public domain books that are just waiting to be adapted.

🧭 Where this fits into my life

Storylearner is one of several things I’m exploring right now. After years of full-time work, I’ve intentionally made space to pursue projects that feel meaningful and energizing. I’m interested in learning, in building tools that support human growth, and in seeing what’s possible with new technologies.

I’ve already started exploring another interest area in parallel — but storylearner isn’t “just a little project.” It’s a serious bet in my portfolio of bets. One I believe in. One I want to see succeed commercially, not just creatively. I’m proud of what I’ve built, and I’m excited to see who it resonates with.

🫱 What next?

I believe storylearner is valuable — now I need to find the people it’s for.

The core experience is live: adapted books with interactive reading support and audio, designed to make language learning through stories more fun, more sustainable, and more effective.

I’m a developer, I could easily keep building and building. Selling and marketing goes deeply against my instincts, but building without users is a common trap for engineers. So for now I am forcing myself to stop building until I find enough traction — I’m looking for early adopters who see the value and want to be part of it.

My next goal is simple: reach my first 100 sales.

If you’re learning Spanish, Portuguese (Portugal or Brazil), French, or Polish — or know someone who is — try it out. Buy a book, read a chapter, see how it feels, let me know how it goes! You’ll not only support the project — you’ll be part of shaping something that’s trying to make language learning better, through the joy of stories.

→ https://storylearner.app

3 Reflections from a sabbatical after Google

Sat, 15 Mar 2025 10:45:59 +0000

I quit Google at the end of July 2024 and I took a work break until mid-January 2025 (when I joined a startup accelerator full time). It would be a miss if I didn’t reflect on my sabbatical experience!

I will describe what I was up to in another post, but in short it was a mix of traveling and hanging out at my home base in Dublin. Here my 3 main reflections!

Not working turned out to be okay

I was concerned about how I will feel about not having a job and how others will react about me not working.

Why would I be concerned?Financial instability: No paycheck coming in.Lack of structure for my days and tasks and accountability from others: Would I get lethargic and waste a lot of time?Uncertainty about the future: Off a structured path so unclear what will likely happen next.

All of the above could make me uncomfortable. The uncertainty, the fear about the future, the lack of direction…

How was I? Surprisingly, pretty great.

I wasn’t worried about finances, because I have worked on my financial mindset for many years. I had a ton of cash in my checking account, enough to cover multiple years of my burn rate. AndI was resolved to not worry about spending it. I gave myself a generous budget, similar to what I was spending while employed and at times higher.

This is obviously a very privileged position, but I strongly believe that how we feel about money is only slightly correlated with our financial situation. There are many people sitting on huge savings and investments who are afraid to pull the trigger. There are also people with barely any money making big bets on themselves, figuring things out as they go.

Being resourceful is more powerful than just having resources.

Was lack of structure or accountability a problem? No. I am naturally very disciplined and a good planner. Both traveling and sitting at home - I was never bored, always had a lot of things to do and rarely lacked enthusiasm. Occasionally lacked energy due to doing too much cool stuff or being sick.

I trust my body that if I am lacking enthusiasm to do things I either need to rest more or find better things to do. I might not want to do things 100% (like a run in the rain!), but at least part of me needs to want it.

Other people were very understanding of the idea of taking a career break. In their eyes “I deserved it”, in their eyes “I got this” and that shouldn’t set me back.

During this sabbatical I was living on my own. When I stopped going to work I ended up without having people around me by default, which was a new thing for me.

While working I would always have coworkers and I would usually work from the office. During the forced work from home period, I had my partner there.

When you have a job, you have companionship of people physically around you. You have meetings. You are forced to interact, but you don’t have much choice with whom. Most of my career I had a pretty good set of coworkers. I would get along fine with most and I would usually really like some. Sometimes I worked with my very good friends. But sometimes it would be mostly professional.More capacity for intentional socializing

During my sabbatical I replaced the shallow level socialization from work with a deep and meaningful intentional socialization with my friends. I was also more open to making new friends and I had the capacity to do that.

While working, I wouldn’t do a lot of meeting people for deep conversations after work during the work week. I would either do some sport (e.g. climbing) and have some shallow social interactions there or I would need some alone time to relax or process what has been going on.

While not working, I had much more social energy, I would be up to meet for lunches, coffees, dinners and activities. And I was lucky to have many friends to spread this social load over. I had a lot of amazing 1:1 time as well as very fun group hangouts.

My lack of constraints also allowed me to bond deeper with people through travel. I met really interesting people by traveling to cool events (like Zuzalu Village in Georgia) and I went on several trips when I spent extended amounts of time with various friends which really helped me deepen those friendships.

Intentional socializing is much better

While working I would occasionally have those deflationary social interactions - when someone’s lack of energy or enthusiasm would drag me down with them.

My sabbatical gave me much more control of my social environment and I ended up developing my relationships and feeling very connected even though I would spend more time alone.I had more energy for meaningful socializing 1:1, I would be much more proactive in organizing it and the quality of the interactions was much higher.

Connecting when people are busy with work?

A surprising amount of people are available during the work week for lunches, coffees during the work day or breakfasts or just a random phone call.

There are many people who don’t work or have weird schedules, you will start meeting them once you are on a weird schedule.

If not enough people want to hang out with you or have availability to do it, you might need new friends. That sounds tough, but wait. You are in luck, if you are not working, you have much more capacity to explore for new friends and invest in those friendships.

When you travel, there are likely other travelers you could meet. People traveling solo are especially willing to connect.

There are various events you can join, social sports classes, and even coworking spaces (you can “work" on your own project there). Show up, be friendly and exchange contacts, it really starts that simple.

I redefined what productivity meant to me

One of my guiding principles for that sabbatical was “not rushing”. I also deliberately didn’t plan any major projects, if it felt right, I would do it, if I didn’t, fine, I would let it go.

Unreasonable non work expectations

I would consider myself a driven, ambitious and disciplined person. I was used to getting a lot done: my job, lots of workouts, side projects, doing courses, hobbies, the list goes on and on.

I was aware that what I wanted was ambitious, but it also felt achievable. I wanted it all.

Turns out that I have trouble keeping up with all my goals and expectations even if I don’t work.

I am not dumb, I swear. I prioritize, I kill projects and say no to people. But still, I am sometimes overly optimistic about my abilities to keep taking on more hobbies, healthy routines and projects.

It makes me realize that I was operating in a very unsustainable way and I have to lower my productivity expectations.

Chronic stress

During my work life I would often feel stressed and overwhelmed. It wouldn’t be very severe, but enough for me to be aware of the looming tension.

I would also struggle with sleep, especially waking up during the night and waking up too early - often with intense thoughts about work.

I would usually fall asleep fine, probably due to being exhausted, and if that didn’t work very well, I would take melatonin and that would usually suffice.

I thought that maybe it was anxiety? But researching the topic made me realize that it wasn’t anxiety. I didn’t have a worrying habit, I would just feel a lot of tension and unease - and that was stress from having insufficient resources to meet all the expectations I had.

I know, semantics, but for me realizing that distinction was meaningful as it helped me diagnose the cause.

I also had a major breakup that flipped my life upside down. I started my sabbatical a year after that, but I think I was still somewhat affected.

I have developed a toolbox for dealing with stress during my work at Google and it was very helpful, but still my problems with sleep were persistent. I also had various tools and habits to optimize my sleep, which I was using consistently. I tested many of them and I was very disciplined about following the ones that worked well for me.

Funnily enough, after my stress melted over a couple of months, my sleep improved. That happened, even though my sleep hygiene habits got worse.

Deprogramming the “hard work” culture

I come from a family of hard working people and from a culture that values hard work and effort. I have always seen a value in working smarter, not harder. But at the same time, in my head, you should still work kinda hard, just use your smarts to get X amount more impact, instead of scaling down your efforts to 1 / X.

I think my programming goes like this: hard work is a moral obligation. Some people are compensated more than others (sometimes due to luck, not always on merit), but everyone should work hard to be contributing. It’s like fairness of effort based on everyone being obligated to give their best. Slacking off when other people are working hard is unfair and immoral.

Of course the reality is much more complex than the above!And this is just one set of beliefs. Another set of beliefs could be that I have one life and I should do the best I can to enjoy it. Working and helping others can be a source of meaning and satisfaction that can be one of the sources of my happiness, so I can do that for that. Otherwise I should use work as a means to get enough resources to enjoy life. If I gathered enough money, then it means that I produced enough value for that time. No need to earn my keep everyday.

Internal productivity

Recently I was watching a lecture about writing by Brandon Sanderson and he said something along the lines of:“You are the main product of your writing, practice of writing changes you and makes you a better writer, a different person”. So it doesn’t matter that you will write books that won’t be published. Do the activity for how it will change you.

I found it very compelling.

Ok, it is kinda obvious. You learn skills to have skills - it changes you. You work out to change your body (or keep it up), it changes you! Sometimes you do that to achieve an external goal like impress others or get a job. But it’s more interesting if it’s purely for you.

Internal productivity is when you are the main product of your activities, when they change you.

And it doesn’t have to matter to anyone but you!

My sabbatical was full of internal productivity.

Quality of life vs Quality of the experiencing of life

I think that everyone will agree that ‘quality of life’ matters. But you might be living a “high quality life”, but be bad at experiencing it.

Rushed, detached, reactive, struggling to be present. We’ve all been there.

I have a strong desire to be good at experiencing my life. I want to be present in my experiences as well as process them afterwards and I want to understand both myself and the world as much as possible. I want to live an examined life.

“Oh, this person must live an examined life” - said no one, ever.It’s hard to pinpoint if people take time to do the internal work that is required. Maybe they are simply not sharing it? But I think it catches up to people.

And all this examination takes time! Time that could be spent on more grind that could be yielding visible results! But I think that I worth it, and I will treat it as productive.

So even though I had many days that were neither “productive” or “instagrammable” or “full of stories”. I will never regret time spent on:

Being in touch with myself, my feelings, thought patterns and opinions
Deep conversations - prioritizing them and trying to improve at them
Learning and reflecting

Conclusions

It felt crazy to quit. But I am glad I did. I believe in closing a chapter to start another one. But sometimes you need a break between adventures. And this was a very good one.

It turned out to be ok to not work for a while and have an unrushed and unproductive period. Even if by the end of it I redefined what productivity meant to me!

I had a lot of fun and intense adventures, but I am really glad for the “boring moments”.

How to transcribe long audios fast with open source (colab included)

Mon, 14 Oct 2024 00:00:00 +0000

In this post I will give you code that you can run yourself in Colab (or on your own machine with a GPU), that will allow you to very quickly transcribe long audios in many languages.

The setup uses the whisper model from huggingface, Google Drive and Colab.

I used it to transcribe multi hour podcasts within a couple of minutes (about 1-5 minutes depending on the length).

Motivation

Some time ago I was getting frustrated that the podcasts that I was listening to didn’t have transcriptions available.

Let’s pick this one as an example - it’s a massive, almost 3 hour long podcast by Huberman. It’s knowledge packed!

Could I get the transcription? It’s not included anywhere, but I could make my own.

Sure, there are services for it, one that looked promising to me is listen411 - it charges 0.06 USD for 1 minute of audio summarization + 1 USD per file, interesting!

A podcast like one I was interested in would cost me about 13 USD, because it’s a very long one. I probably listen to 5 podcasts per week or more, so let’s say 20 per month. Ugh, the costs could add up fast.

If you go one level down, there are also APIs from cloud providers, which could be a good alternative. I checked the costs of transcription in google and aws: 0.024 USD or 0.016 USD per minute respectively, not bad, but it’s still about 1-2 USD per long podcast.

So I wondered, can I do better myself? Based on my knowledge of the current state of the ML, audio transcription is a pretty much a solved problem and there are excellent models available publicly for free. With open source libraries and Colab (easily accessible GPUs) I could build a DIY solution that would be much cheaper.

Cost of running the DIY solution

The costs of the DIY solution are pretty much just the Colab running costs. When you break it down, it’s about 3 orders of magnitude cheaper - provided that it’s a fast process! It can also be totally free if you use the free tier of the Colab service.

Colab has a free tier that I think could be sufficient for that, but if you want a more guaranteed access to resources, you can use the paid version of Colab as well. the Colab GPU pricing is 0.196/hr USD (T4 as of 2024).

A cost of ~(one podcast) transcription - ~3minutes GPU time (T4) - approximately 0.01 USD.

And if you were to scale it, you could also run on (self) hosted GPU driving the cost even lower.

Obstacles on the way

The code that I got at the end is very fast and provides high quality results, but it took some trial and error to get things right…

Major issues I encountered were slowness and poor quality of results.

I hate slow running models, I have no patience for long running pipelines - it also gets expensive fast as you pay for the computation time.

My plan was to use huggingface (as it’s a great ecosystem) and I used a tutorial with speaker diarization as a starting point. Dirarizaton lets you annotate the transcription with different speakers and when they speak automatically. It appeared relevant and promising.

But in practice copying the approach with diarization wasn’t feasible due to slowness for audios longer than 5 minutes. It just seemed stuck for hours without showing much progress. I spent quite a long time waiting for the transcriptions.

I also spend a good bunch of time fiddling around and learning API details of hugging face libraries, that tutorial confused me more than it helped.

I wondered if the models I was using were too expensive to run and I played with different models trying to assess if I can get my transcriptions faster, and some results weren’t the best…

In the end, the breakthrough resource that helmed me was studying the Insanely Fast Whisper repository.

Especially this notebook.

Insights from that notebook were enough to simplify my code greatly and speed it up dramatically!

DIY solution

The setup

The overall solution I built and I’m sharing is:

upload audio files to a personal Google Drive Directory (either manually or another script, e.g. RSS feed client) - the directory is called audiotranscriptions
mount Google Drive in Colab and scan the directory for audio files missing a transcription file
transcribe audios that needed transcriptions and save transcriptions as txt files in Google Drive

Transcription logic

Without performance optimizations the code is as simple as:

import torchfrom transformers import pipelinepipe = pipeline("automatic-speech-recognition",                "openai/whisper-large-v2",                device="cuda:0")filename = "./myfile.mp3"transcribed_text = pipe(filename)

As the pipeline is so simple and it uses whisper (state of the art model!), it’s already pretty good, but it can be improved.

using half precision - this will let the model use less GPU memory
chunking and batching - chunking will dramatically decrease complexity as the computation time usually is usually not linearly proportional to the duration, but a higher order polynomial. Chunking puts the limit on that time, it’s like processing a bunch of smaller audios and stitching them together. Batching helps to parallelize that. Additionally batching goes hand in hand with using lower precision as if a single batch takes less memory, you can have more batches at once.

The code modifications are very simple.

Enabling half precision:

import torchfrom transformers import pipelinepipe = pipeline("automatic-speech-recognition",                "openai/whisper-large-v2",                torch_dtype=torch.float16,                device="cuda:0")

Enabling batching and chunking:

filename = "./myfile.mp3"transcribed_text = pipe(filename,                        chunk_length_s=30,                        batch_size=16,                        return_timestamps=True)

Helper logic

The rest of the code from the attached notebook glues it all together. Here are the descriptions of the main parts.

Mount the Google Drive and navigate to the directory audiotranscriptions containing files needing transcriptions:

from google.colab import drivedrive.mount('/content/drive', force_remount=True)%cd /content/drive/MyDrive/audiotranscriptions

Utility functions for the logic if the file needs a transcription:

from pathlib import Path, PosixPathdef get_file_prefix(filename: PosixPath):  return filename.name.split(".")[0]def needs_transcription(audio: PosixPath, transcriptions: set[str]):  return get_file_prefix(audio) not in transcriptions

transcribe_file is calling the previously created pipeline for a given audio filename and saves the transcription under transcription_filename.

def transcribe_file(pipe, filename, transcription_filename):  print("transcribing", filename)  outputs = pipe(filename,                chunk_length_s=30,                batch_size=16,                return_timestamps=True)  text = outputs["text"]  with open(transcription_filename, "w") as f:    f.write(text)def transcribe_all(pipe, audios, transcriptions):  for audio in audios:    if needs_transcription(audio, transcriptions):      transcription_filename = audio.with_suffix(".txt")      transcribe_file(pipe, audio.name, transcription_filename)      transcriptions.add(get_file_prefix(audio))

Putting it together:

scan the directory
find audio and text files
transcribe whatever is needed

files = Path.cwd().iterdir()audios = [file for file in files if file.suffix in [".mp3", ".wav"]]transcriptions = {get_file_prefix(file) for file in Path.cwd().iterdir() if file.suffix == ".txt"}transcribe_all(pipe, audios, transcriptions)

Code & results

You can see the colab notebook here.

I have tested the script on podcasts using English as well as Spanish and Polish and the transcriptions looked very good to my naked eye. I haven’t done a deeper quality assessment.

What’s next?

After you have the text, the fun just starts!

I have further used LLMs to strip the transcripts from promotional content, make summaries and extract wisdom out of the longer audio. But that is a topic for another article!

I killed my side projects - clean slate

Sun, 15 Sep 2024 00:00:00 +0000

I have recently shut down the indie products I’ve been accumulating over the years, RIP:

redeal.app - real estate investment calculator - inspired by bigger pockets, initially scraping daft (irish real estate listing website)
invertimo.com - investment tracking and bookkeeping (open source), useful for stocks, funds or any sort of crypto assets if multiple exchanges/etc are used, was integrated with degiro, interactive brokers and binance… + manual uploads
watchlimits.com (that is partially alive as the last version is still available in the chrome store, but the ‘premium features’ won’t be working) - a productivity extension helping with excessive video watching online

I lost motivation to work on these projects, they were either in zombie stage (Redeal/Invertimo) or in somewhat alive stage (watchlimits), but little momentum or financial potential. I would occasionally get some support/feature requests for watchlimits, but I didn’t want to throw good money after bad (in terms of my productive project time).

And there were also server costs and upkeep costs of my time to get everything running smoothly and securely. Additionally there was guilt from not providing a good product experience as I wasn’t motivated to deliver fixes or improvements.

So I pulled the plug! Servers turned down. I didn’t bother in open sourcing closed source stuff or selling the projects.

Joy of creating vs burden of maintaining

There are people who are great builders and there are people who are great operators (or maintainers or gardeners). I am a capable operator, but it has to be related to something that I really believe in. I’m not internally motivated by the activity itself. While I can create stuff, just for the joy of creating.

I had a pretty strong creative impulse to build side projects over the years. I had a dream of having my software company for some time, but it’s a pretty difficult task. I also didn’t have very good opportunities to build products e2e at work, so building complete products as side projects was a good outlet. It was a nice way to experiment with UX, tech, product design, marketing - all on the side of working in a pretty demanding job.

And I was somewhat successful, because I delivered 3 project MVPs - Redeal (real estate investment), Invertimo (stock investment tracking and book keeping), Watchlimits (productivity chrome extension). All got some users who found value in it. With watchlimits getting some payments and reaching almost 1k of free active users (and more installs).

I learned quite a lot about building a product, getting feedback, marketing, I got confidence in the fact that I can build stuff - even if part time and it could be better it was good enough to execute the vision, the technical side was never a blocker for my projects.

But what these projects had in common was that they had very little potential or momentum to become profitable. I also wasn’t that passionate about any of the topics. The itch was scratched, the products remained alive, but they were pretty much in zombie mode.

Why didn’t I shut down the projects the moment I stopped believing in them?

Cleanup is work and it’s somewhat tedious
Emotional attachment

To properly unwind the project I would have to put in quite a lot of hours and I am chronically overscheduled - even if it’s a spontaneous week of windsurfing - it’s hard to fit in a bunch of productive project hours for a thing that I don’t really want to do. It’s very easy to procrastinate.

And there is an emotional load of killing your baby, even if you stopped caring about the baby - you feel bad about both not caring about your baby and then killing your baby project.

Clinging after stopped developing

Some thoughts that were at the back of my mind:

“Maybe it will still work out?”
“Maybe I will feel like working on it again?”
“It’s proof that I can build things and release things, right?”
“What will people who observed my journey think?”
“It’s not expensive to keep around… some people have value from it”

I never had any solid conviction in my project watchlimits. From the start it was “let’s do this as a learning project and carry on working on it until I feel like it or find a better thing to work on”. It started during a hackathon. I had a strong desire to have a project to work on, but finding the right idea was hard, talking to prospective customers was hard, so I did what I knew how to do… jumped into the building process.

Doing back of the napkin calculations only confirmed my hunches. It was very unlikely that the project was financially viable. But I tried! I tried a lot of stuff and I learned pretty well how it feels to release a product with little financial traction.

After I couldn’t really convert people to paid, I made the ‘premium’ features free. It was somewhat disheartening that after I put effort into building the premium features almost no one was using it. After opening the features, people did actually find value in them and it was making me happy for a while.

And the user base has grown roughly from a hundred to about 1k active users and they started giving me more and more requests and at some point I just stopped working on it entirely. My heart wasn’t in it.

Moving on

It feels good to have a clean slate. The limbo of “it’s still somewhat alive, but I don’t want to work on it” was taking a toll on me. I want to spend time on projects that I find interesting, exciting, or are related to some important goal to me.

I am still happy I build those side projects, I got a lot out of it:

Experience - knowing that you can do something and having feelings of competence are very important - they reduce the activation energy for starting new projects
Codebases - I can look up old solutions to common problems - having boilerplates is very valuable as it can make me go from idea to production faster
Blog posts - some distilled knowledge and signal of expertise

Now without the small infrastructure burden and a slightly bigger psychological burden, I am freer to pick up new stuff. Wish me luck!

Learning Deep Generative Modelling from Stanford

Fri, 30 Aug 2024 00:00:00 +0000

I am a big fan of self-directed learning, but I have never before spent almost 2 thousands of dollars on a single course. Was it worth it? Who could most benefit from that?

I took the Deep Generative Modeling course from Standford online for 1750 dollars.

It was a first such course for me and before committing to it, I had some questions and doubts as it was rather expensive. I hope sharing my experience will help out people in a similar situation.

You can benefit from this post if you:

are interested in learning about AI/ML or specifically generative modelling and you are wondering if XCS236 is a good course to take.
wonder if it’s worth to spend almost 2k usd on a course, vs 50 usd/free on other platforms/DIY approaches.
want to know how to best take advantage of such course if you were to invest in it.

My background

Coming to the course I had been working in tech for 10+ years in software engineering positions and had a degree in Applied Physics. This meant that I was not afraid of math (calculus especially) and was very comfortable in coding. My math base was somewhat rusty, but I compensated by taking some Coursera courses to refresh the topics I needed more confidence in.

I also had some knowledge/experience with ML/AI - I took many courses along the years even though my job experience with that was very limited. I also have done some side projects that required using models.

My knowledge of generative modeling was mostly related to TTS models and content of Dive into Deep Learning and Deep Learning by Deep Mind and UCLA.

Course content and format

The course was focused on fundamental math and techniques in Generative Modeling:

Autoregressive models
Variational autoencoders
Normalizing flows
GANs
Energy based models
Score based models
Diffusion models

You can see the full syllabus here.

The course gave a deep understanding of probabilistic modeling across all the approaches and taught me a bunch of math tricks. The assignments helped me to better understand the math and as well as actually implement all the approaches on toy datasets.

The concepts were explained in a way that built on each other and gradually expanded the student’s arsenal of techniques.

The assignments were rather math heavy, but would really help with better understanding the nuance of the theory material which I later really appreciated. The coding parts were well done and focused on the core of the algorithms and math and provided good scaffolding for the essential bits to make things work, but not specific to the content of the course. The code was all in pytorch which is what is commonly used in the industry. The assignments focused on autoregressive models (including transformers), variational autoencoders and GANs. There weren’t any assignments related to diffusion modeling which was disappointing.

The course material was up to date on the latest diffusion based approaches. The main professor of the course Stefano Ermon is one of the top Generative Modelling researchers (H-index 84 as of 2024) with a specialty in diffusion modeling.

The course was not focused on transformers, LLMs, or language processing in general, if this is what you want to learn about, this is not the right course.

Platform and format

The course was a mixture of pre recorded lectures broken into chunks on specific topics and a set of assignments that contained coding and written parts. It’s 10 weeks, but as it’s self paced it could be completed earlier.

Lectures were hosted on the stanford online platform (also available on youtube, but not unprocessed), the assignments were in private repositories on github, but you would submit the solutions on the stanford online platform.

There were 3 assignments and no graded projects. The assignments had pretty generous deadlines.

I believe that about 400 people participated in my course cohort. There was a community on slack for students on slack, and each student would also have a course facilitator, students would be encouraged to form groups on their own. There were regular live zoom meetings for office hours with the professor/TAs, show and tell and guest speakers as well. Course facilitators were available for 1:1 video chat or chats on slack.

What I enjoyed in the course

Slack community for students and TAs/Course Facilitators
Study groups - where I could chat with other students and meet with them virtually
Additional resources for students
World class faculty - Stefano Ermon is one of the lead diffusion researchers!
Homeworks were difficult enough to motivate a lot of learning - lots of math proof1:1 time with course facilitators
Deep and well organized understanding of the models aided by theoretical homeworks and practical exercises
Well scaled down problems - no need for GPUs
Lectures also available on YouTube (convenient for watching while doing house chores :D)
Alumni slack and private linked-in group - this was a nice surprise at the end of the course. Based on the introductions on slack, there were very many cool professionals from all sorts of companies attending the course, so this is a goldmine of high quality contacts.

Money on the line helps with motivation

I start a lot of courses and I only finish some, it takes quite a lot of persistence and motivation to stick to them.

The fact that I spent 1750 dollars on the course made me really want to complete it. Just imagining that I could fail it and waste all this money would make my stomach churn. So I really put in the work. I prioritized time for the assignments, I started early and I studied additional resources. It motivated me to revise my math skills and read additional papers.

The level of math in the first assignment was quite a shock to many of the students (including me!), but as I persisted I got into the groove of writing math proofs. My comfort level got expanded for sure.

That said, there was a window of time to give up on a course and get a refund or transfer to a different course if the content wasn’t matching the student’s expectation.

You get what you put into it

As the course was hard, just completing it provided a lot of value, because it required understanding of the topics. I completed the course with more than 100% of the points (thanks to assignments for extra credit!), but I still think that I could have taken more advantage of it!

What did I do right?

Some of the things I did right:

Learning from additional resources to fill in my gaps and enhance my understanding
Using Spaced Repetition (ANKI) to better retain the material - since I put all this effort, it would be a waste if I promptly forgot it
Choosing to do (most of) extra credit assignments
Meeting fellow students - I was a instigator of several discussions and video calls, people were pretty interesting
Contacting the course facilitator when stuck

What should I have done differently?

I would have benefitted more if I did the following:

Picking a project to do before the course even starts and partnering with other students to get it done - this was the part that was different from the live stanford course - I think it would be provide a great excuse for a intense learning and collaboration and also taking better advantage of course resources (e.g. questions to professors and TAs)
Taking more advantage of the course facilitators - e.g. please double check my code or reasoning on a proof. It would save me from wasting time.
Collaboration with other students and taking advantage of networking
Take more advantage of live zoom sessions - I didn’t attend most of them and rewatched some of the recordings.

That said, sometimes life gets in the way and it’s hard to maximize every opportunity, so I won’t be too hard on myself about it.

How to do a DIY alternative

If you are a sufficiently motivated and well organized individual with lots of time on your hands, you could recreate most of the value of the course doing the following:

Watch the youtube videos and read the lecture slides (syllabus)
Read the associated materials
Implement generative modeling using major model approaches on toy datasets such as:
Play with state of the art models from hugging face:
- Transformers
- Diffusion models
Devise and work on a project related to the topic

But what about the help of facilitators and the community?

There are ways to get around that too, for example:

Use discord, e.g. ML/AI communities, like one around hugging face - to meet real people and get help.
Use your network to get skilled people to help you, you might actually know people who know a lot about AI/ML. People like to help smart and motivated individuals on the topics they are passionate about. If you don’t know such people, go to some local meetups.

Conclusion

I found the course very valuable for me, but it’s not a course for everyone and you can also learn a lot on your own.

I was interested in the content and happy to put in the time, additionally my work sponsored 2/3rd of the cost of the course. High cost motivated me to prioritize it. And a Stanford credential was an added bonus.

But I would not immediately generalize the worth of that specific course for me to all such courses, even all the Stanford ones.

The factors I would look for to determine if the course if worth it for me:

unique material that can’t be easily learned on other platforms
access to world class faculty
access to the alumni community/fellow students
topics relevant to my interests/future plans and my own availability to take advantage of the course

I hope that helps! Stay curious!

Postmortem of my 9 year journey at Google

Fri, 16 Aug 2024 00:00:00 +0000

I started writing this retrospective during my last week at Google, I have already wrapped up everything, had my goodbyes. In the spirit of SRE (as an ex-SRE), I thought it would be fun to write a little retrospective in the form of a postmortem.

Introduction

I joined Google young and relatively inexperienced and had spent about 9 years there.

I started my journey in software at 19 (first internship) and then continued working part and full time while continuing my degree in Applied Physics. I got disillusioned with working in physics during the course of my degree, software turned out to be a more promising career path.

At some point I got head hunted by a Google sourcer. That resulted in an internship in London when I was 22, later I joined full time in Dublin. I worked in several teams around three products: Bigtable, Persistent Disk and GCE VMs (virtual machines). I include a detailed timeline at the bottom of this post.

What did I expect from joining Google?

After I got a taste of Google during my London internship, I was excited for more. What appealed to me was the level of engineering, how sophisticated and interesting the technology was and the level of engineers.

My only experience until then was with small software houses and startups in Poland, but Google… Google had some of the best tech in the world, and I had an opportunity to work with it and learn it. That was making me very excited.

Google impressed me. I didn’t see myself bored easily, I could stay in for longer, maybe even 5 years and I would still continue learning.Other things that appealed to me were the pay, the perks and fun international community.

From the big picture perspective, I wanted to start my own company, but a detour at Google could give me skills, fun and money to be better positioned to that in the future. I took the plunge.

So, how was it?

Overall, it was incredible. There were ups and downs. Heartwarming moments, joy of teamwork, satisfaction, as well as lots of stresses and frustrations.

It was a mutually beneficial exchange, I poured my energy, soul and cognitive resources and I got form it:

tons of money
engineering skills, low and high level, great systems understanding and world class skills in handling crises and debugging
leadership and management skills
satisfaction from being part of a cool company
cool offsites and business travels
lifestyle perks, including things like pool in the office, world class gym, all sort of sport classes, weekly massages, tasty and/or healthy meals, healthcare on site
great community and relationships

At some point the exchange became less attractive to me. There were several factors:

reached/exceeded my financial goals
fascination with Google tech waned:
industry somewhat caught up
reality of actual work not being that mind blowing
my interests saturated/shifted
desire to start my own company (yes, I can wait, but not for decades!)
company became less fun/cool
budget cuts (less business travel, no mind blowing offsites anymore)
layoffs
Head Count shifting to cheap locations aggressively - less opportunity for me to grow my organization locally
lots of overheads related to security/regulations
many hard engineering challenges due to complex systems and relationships between teams slowing the velocity
cognitive load - this was less of a factor in my last role, but bigger problem in the previous one. Google tech has lots of complexity and nuance, it’s pretty typical that an industry hire employee might need a year to fully ramp up which is pretty crazy
opportunities for personal growth somewhat unclear
there were no dramatic shifts in sight, just more of the same (which is also valuable, I just didn’t want coast)
tapped out career-wise: L6 ICs are pretty rare - it already is a top tier of seniority in engineering, I was not really interested in L7 as L7 is more of a political role than the engineering one on the individual track, and on the management track there was no headcount in sight… if I had an opportunity to become a manager of managers and run a big org under myself I would stay longer
shape of the technical work was not aligned with my interest in what I would like to be growing at.

Lessons learned

It’s hard to compress 9 years of learning into a bunch of bullet points. I learned technical skills, I learned soft skills, I became a better and a wiser person and a leader.

And in the spirit of being a (somewhat) anxious overachiever, I consistently felt that I could be doing better along the way - always motivating me to strive and learn.

What went well

I got promoted quickly
I ended at L6 which is highly regarded and well compensated
I always had lots of autonomy
I could always push back for work life balance (reasonable hours and workload)
inspiring, motivated and smart coworkers
earn more money that I could have imagined
used a lot of perks that enabled very healthy lifestyle
lots of fun travels for work (business and offsites)
grew as an engineer and as a leader
drastically improved my soft skills
made a lot of work friends
learned about lots of cool technologies
had an opportunity to create my own team from scratch
working 60% or 80% were fantastic for my lifestyle and building relationships outside of work
learned a lot of tools for dealing with (chronic) stress

What went poorly

I have overstayed as SRE - Dublin didn’t have much options and I didn’t move to a different site (inertia, personal reasons, etc)
oncall was stressing me out and disrupting my sleep,
not the best fit for my personality which is more optimistic and creative - this led to feeling of misalignment, I felt somewhat unfulfilled at - work in terms of type of work and projects and compensated outside of work with side projects
24/7 nature of the organization was making it hard to disconnect
USA centric culture, if you are not in USA at Google and don’t have a big presence in a location it’s a bit like swimming upstream, it’s easy to feel isolated, sidelined or on the flipside overwhelmed with late meetings
Promised HC not landing - e.g. I was promised further expansion twice, which later was scratched and then promised again in a bit different form…
senior level managers overwhelmed and not providing support/feedback or pretty much any oversight (there were moments when things felt a bit like a wild west)
there were many periods when I felt like I was overwhelmed with meetings/repetitive work/underresourced team, but at the same time not having good engineering/or management growth opportunities
cognitive load at Google is very high - there is a countless number of systems and technologies that it’s useful to have in mind and can affect your system in one way or another (this is especially bad in SRE)

Where did I get lucky?

Google stock did very well, additionally with my rapid career trajectory, I did very well too 🙂
The people I hired turned out to be really solid
I somehow figured things out - building a track record of being high performer even though some things were definitely stretch opportunities
I made good financial decisions, could have been luckier, but the decisions had good thought process
I made amazing connections

What could I have done differently?

Move out of SRE earlier since from the beginning I knew it wasn’t what I wanted
Move to a different site - it’s easy to cling to a thing that is already good, there is a cost of switching as well, but I undervalued exploration historically
Take more advantage of education reimbursements (e.g. take more stanford online courses)

Action items

A typical postmortem would present a table of categorized action items. But I’m not here to mobilize anyone…

The next thing for me is a sabbatical consisting of at least 6 months on exploring, relaxing, learning new things and expanding my idea of what I could be doing next.

I have a tendency to underexplore and I love being productive with a clear objective, so a sabbatical is a psychological challenge. I will be writing more on the topic so stay tuned!

Timeline

Summer 2015 - App Engine SRE Intern (London)

Cloud Bigtable SRE Era (Dublin)
joined as an L3 engineer
got promoted to L4 within 9 months
Cloud bigtable TL role
got promoted to L5 within 1.5 year

Persistent Disk SRE Era

joined as an L5
wondered around until landed in a work group related to project that later became hyperdisk
the US sister team disbanned and to be recreated in Seattle
became a defacto SRE TL of hyperdisk and created a team for it
promoted to L6
became a manager of Persitent Disk IO SRE
Persistent Disk SRE issues with burnout and attrition (SRE hiring issues, rapid dev org growth 40→200, etc)

GCE Fleet Maintenance Dev Era

joined as an L6 IC SWE
director who hired me became a VP and dumped me on some other manager that was overloaded
took over a struggling internal project and made it successful and recruited a team around it in Dublin
transitioned from IC to a manager of 4 and then 6
started a 2nd team under a different senior manager
defragged the 2nd team because of the lack of head count, reinvented the main team charter

I quit my startup because of comic sans

Fri, 14 Jul 2023 00:00:00 +0000

Okay, it wasn’t really caused by the Comic Sans.

I had very fun 3 months of pretty fun collaboration and intense learning, but in the end I had this very clear feeling that I would be better of on my own.

It wasn’t awful, but it wasn’t great. There were many small and big things that were making our fit non-ideal, and I had doubts growing in me,and one Sunday night, a pick deck in Comic Sans topped me over the edge to call it quits.

Company vision fit

Both my partner and I wanted to build a profitable company that would be bigger than just the two of us. But he was extremely focused on getting investment, when I was more interested in retaining control and bootstrapping as much as possible.

It makes sense, I’m practically self sufficient - I have lots of money saved and invested, skills that could easily give me money from contract work and skills to build and (even) market the product.

On the other hand my cofounder was more dependent on others, had limited runway, higher monthly burn, and much weaker technical skillset. It made a ton of sense for him to be securing funding.

Funding company with venture investment can be pretty useful:

there are some types of companies with high startup cost that are not feasible for a typical ‘bootstrapper’
acceleration of growth - no one wants to go through a slog if you could make the growth faster
a blanket of security before you reach profitability

But it has a cost:

You give away equity
You put time into securing funding instead of making a useful product, it takes a lot of time into making pitch decks, getting known around, etc.
it shapes the type of a business - ideally, it would be a ‘unicorn’ and investors will push you into this direction even if it makes the business fail in the end

I’m not sure what I want exactly, sometimes I think that I want a lifestyle business, sometimes I think I would like to make it big, but in either case I want it to be my decision and not due to pressure from investors.

Work style fit

I found my cofounder work style to be chaotic. We would make a plan, he would pick a lot of items to do, but then everything would be in progress, but hardly anything really done unless there was an external deadline, like for an accelerator application. We would agree for a system for tracking tasks, but then he would not use it for the most of the time. Was that necessarily a deal breaker? Not really, people can have different styles andstill get work done. I wouldn’t force people who have many years of experience to change their work style just to suit me, but it also didn’t make me feel too good about it.

And there was also an aspect of attention to details. Which was not my cofounder strong suite and that was leading to a bit of frustration and rework on my side. I wanted to build a high quality product and that is at odds with low attention to detail.

The issues above don’t make my cofounder a bad entrepreneur, I think that a dynamic, slightly chaotic work style with lots of work in progress could work for some people, but I just found that I prefer people who are more methodical and steady.

Skill fit

I brought to the table very strong development skills, ability to execute quickly on the product. Thanks to my earlier product building adventures I also had plenty of ideas about idea validation and marketing and some experience with content marketing, SEO and paid advertising.

My cofounder brought the idea and the ability to talk to customers, many years of tech/consulting adjacent experience and “street fighting”. Ideally, he would also bring in marketing and design skills, but he was pretty green to marketing and not particularly design inclined.

With a non-technical/technical cofounders, the typical split would be to have a CEO and CTO, which would make me the CTO and give other important tasks to my partner.

However, what makes me interested in entrepreneurship is handling all those CEO things. I’m actually interested in business. I wouldn’t want to delegate those aspects of the entrepreneurship. That combined with my partner being new to this, made me resistant to this partnership.

Friendship fit

Turns out we had very, very little in common. Sure, we were focused mostly on working and the business, but during the time we were hanging out I haven’t discovered a single shared interest apart from wanting to start a business.

And even though we lived in the same city, it was quite far apart. And we had different lifestyles - he was a young father, living in the suburbs, me - tech person leaving in the downtown, doing many sports, etc.

On top of that, we had differences in the background. I’m pretty much a nerd: science, hacking, technology, fitness, finance. He was more of a businessy-consultant “people person”. And we would have differences of opinion based on that.

Idea?

Was the idea a killer idea? The jury is still out, it was a very interesting project from the technology perspective, and I could see it working from the business perspective if we executed it very well. It wouldn’t be a particularly easy project, but it had potential.

But at the same time, I think I can do better in terms of difficulty of making it a successful business and skill fit for me. I don’t have a better alternative right now, but I will keep looking and there is always an option to start a boutique software solutions agency which to be honest also sounds like a lot of fun and an easier business to bootstrap.

It was pretty difficult to abandon it, I got really invested in building it, there were more things I wanted to finish, but it had high opportunity cost and working hard on it was wearing me out.

What’s next?

After 3 months of intense learning and building while also working my day job I am going to be taking a bit of a break and instead work on non-critical side projects. I will also invest time into researching and validation of some new business ideas, but probably won’t jump full on into implementation.

Did I make a good decision? The time will tell, I think that it was right to go with my gut, because likely these many small problems and doubts would only get bigger with time.

Going on my own with the next project will for sure be hard and lonely. But I have the indie community and friends, it won’t be too bad!

Everything being just on me is a bit scary, but I did that in the past, I am very independent and I like to be in control. I got this! I’m not giving up on my entrepreneurial dream!

Two worlds: idea people and developers

Sat, 29 Apr 2023 00:00:00 +0000

I recently attended a startup weekend and was surprised by the backgrounds of the other attendees.

As a software engineer myself, I expected to see mostly engineers, but that wasn’t the case. Instead, I found myself surrounded by people with a diverse range of backgrounds and experiences, all of whom shared one thing in common: they had an idea.

At the same time, couple months ago started running an Indie Hackers Meetup in Dublin, a software startup meetup that mostly attracts developers.

And from other breaking news, as a result of that startup weekend I’m now deep into a new project. For the first time joining forces with a cofounder. The project is still in pretty early stages and I don’t know if it will become a serious business yet. Fingers crossed.

This all got me thinking about the two distinct groups of people in the startup world - those with the skills to execute an idea but lacking a good business idea that would solve a real life problem and had financial potential, and those with great ideas but lacking the technical expertise to bring them to life.

How can they meet and should they join forces as cofounders?

Two groups of people

Many developers dream of starting their own software businesses, while countless non-developers struggle to find the right development team to build their ideas.

Do they need to join forces to build a company together? Well, no. There are many other ways.

A developer can still get hired to work on another person’s idea and the dreamer with an idea can seek funding through an angel investment, family and friends or VC.

But the truth is, these approaches have some serious drawbacks.

For developers, it often means giving up control over the direction of the company they’re working for, in exchange for a relatively small amount of equity.

And for entrepreneurs, it can mean giving up a significant chunk of their ownership in order to secure funding. And even then, they may struggle to find the right talent to build the technology they need. In many cases, they end up working with developers who don’t share their vision or passion, resulting in subpar solutions and missed opportunities.

So, should they join forces directly then? In some cases, I believe yes.

Benefits of bringing the idea person and the developer together

For the developer, partnering with an idea person means they can work on a project they are passionate about, have more control over the direction of the company, and potentially own a larger percentage of the business. As a result, they’ll have a vested interest in the success of the company, and are more likely to go above and beyond to ensure its success.

For entrepreneurs, working with a skilled developer can help them build the technology they need to turn their idea into a reality, quickly and without breaking the bank. Instead of struggling to find the right talent, entrepreneurs can rely on a partner who shares their vision and passion, resulting in better solutions and increased opportunities for success. By partnering with a developer, entrepreneurs can also reduce their dependence on external funding, potentially allowing them to bootstrap their business and maintain a greater degree of control over their company’s direction and growth.

Through my indie hacking adventures, I realized that building a profitable product can be actually a very emotionally hard and lonely journey. Having a cofounder means that each person can support the other through the ups and downs of building a business. They can share the workload and take on different responsibilities, leveraging each other’s strengths to achieve success. This mutual support is crucial when it comes to staying motivated and pushing through the difficult times that inevitably arise in any startup journey.

This all sounds pretty great. But are ‘just an idea’ or ‘just dev skills’ enough to partner up? I think that partnering up with anyone shouldn’t be considered lightly.

More than just an idea

While having a great idea is a good starting point, it’s not enough to make someone a valuable cofounder. Yes, ideas are important, but I believe that for startups, the real value lies in the execution and ability to iterate.

Think about it - the likelihood that the first version of the idea will work is incredibly small. It will likely require a lot of iteration and pivoting, and the end result may be something completely different than what you originally envisioned.

Don’t bring someone as a cofounder if they just contribute an idea, but don’t have much else to offer.

More than just development skills

Similarly, bringing up just skilled development labor is also not enough to make someone a valuable cofounder.

Some companies don’t require extensive development work. And even if technology is critical to your business, simply having a skilled developer on board is not enough.

Early stage business often require a lot of non-coding work. And if the business succeeds, the startup will need a CTO, not just a developer and that is a very different role requiring different types of skills.

Cofounding is a long term commitment

Even if you bring considerably more than just idea or just the development, and are both excited about the project, before you commit to such a partnership, it’s important to consider the long-term implications.

As a co-founder, you’ll be investing time, effort, and potentially money into the venture. You’ll need to share the vision and values of the idea-non-tech co-founder, and be willing to work together to build the company, not only when times are great, but especially in crisis, when things are not working.

A successful startup requires a diverse set of skills, such as business strategy, marketing, and customer development. You and your co-founder should complement each other’s strengths and weaknesses, and have a clear understanding of your roles and responsibilities and over time develop strong mutual trust.

If you are not sure that your potential co-founder is ready for that sort of commitment, don’t become co-founders. Or start with a limited time mini-project first.

There are other ways to take advantage of the complementary skills.

Taking advantage of the other camp without co-founding

Not everyone is ready to commit to a long-term partnership, and that’s okay.

There are other ways to collaborate and take advantage of each other’s expertise without co-founding.

As a developer, you can go to startup events, meet idea people and collect a wide range of ideas there.

And as an idea person, you can approach developers to get their input on your ideas and even hire freelancers to build a minimum viable product. This way, you can still benefit from the knowledge of both camps without the burden of co-founding.

Conclusion

Overall, joining forces can be magical. With the right partner by your side, you can achieve more than you ever could alone, and build a company that you’re proud of.

But it’s also a big potential risk to partner with anyone as cofounders and there are other ways to utilize the complementary skillsets and perspectives.

Regardless if you want to commit or not, I think it’s worth to explore the other camp. Go to developer events as a business/idea person and go to business events as a developer.

I strongly believe we could benefit from more cross-pollination.

Fun indie projects

Mon, 27 Feb 2023 00:00:00 +0000

I wrote in November that I’m taking a break from indie hacking, well, the break didn’t take long (2-3 months).

A break made sense at that time, because my project watchlimits.com wasn’t getting traction and I just changed jobs. New job presented a lot of new learning opportunities and as any change was somewhat stressful, so I didn’t want to overload myself and risk a burnout.

I am back in a not fully “serious” capacity. I am not doing any serious business right now. I have a bunch of promising ideas, but for now I’m exploring some pretty dumb ones.

I’m working on projects that I don’t plan to monetize and for which the main goals are having creative fun and learning.

Fun and learning » MRR

People have different motivations.

Often it’s about making money. Money can be a great motivator or a bad one. It’s a great one when it’s aligned with making something actually valuable for others. But it’s a bad one if you feel like it forces you to make the wrong choices.

There are many valuable things in life, we should focus on the ones most important to us in the moment. I have a full time job, which pays me well, but constrains me in terms of what I do and how I do it.

Being independent allows me to do things that don’t earn money, just because I want to do them.

So right now I am focusing on having fun and learning new things through my side projects. And I am serious about this. Fun and learning are the main goals, a little challenge or discomfort is fine, but no sustained grind.

My current priorities:

being creative & playful
building my craft skills - variety of stuff I can build and speed of prototyping
shipping & sharing stuff

I would like to be able to do it forever. I think that the person who has the most fun wins and is least likely to burn out and give up.

It doesn’t mean that I will on purpose not try to make any money. Making money can be fun!

But I’m also ok with doing projects that have no monetary prospects at all and are just a fun, creative expression.

New projects

Expect new projects to spring up like mushrooms 🍄🍄🍄.

I’m ok to start a lot of things and to also abandon a lot of things if they don’t spark joy anymore.

🐢 Focus Turtle

Focus Turtle is a chrome extension that helps people focus by reminding them about their goal when they get distracted.

Stack:

React
TypeScript
Tailwind CSS
Vercel

Goals:

do something cute and fun - creative outlet
experiment more with the browser extensions
get better at design, react, typescript, ux
another digital asset that brings people into my orbit of my project ecosystem
make something useful for other people
have fun!

Non goals:

making the focus turtle into a commercial product

With these goals in mind:

I can spend time to do things right, there is no rush
It’s ok to waste time being “inefficient” - I’m really bad at something it means I am outside of my skill comfort zone
Things that push me into new directions / are a practical application of my skills win
Don’t do things that feel like a grind

💓 Beat Focus

Beat Focus is a mobile app that let’s user play binaurial beats on a timer in a background when they need an extra focus boost.

Stack:

React Native
TypesScript
Expo

Goals:

I wanted an app like this as I like focusing for short bursts of time and would listen to binaural beats myself
learn to develop apps with React Native
Make something quick and simple but extensible

🏠 Aer Teach

Yet another Chrome Extension! This one was a group project developed during Dublin Air Quality Data Hack hackathon in late Feb 2023.

The idea there is to let you see pollution data right when you are looking for a place to buy or rent based on extensive new air quality data from Air Street View dataset.

So the extension is integrated with Daft.ie, the biggest real estate website in Ireland, the extension will display the relevant info, link to additional places to learn more every time user browses the listings. This is useful, becausepeople might remember once to check the data about air quality, but forget the next time as they likely won’t have a habit. On top of it, it saves a lot of otherwise tedious work.

The pollution data is processed by a python backend behind a simple API.

We haven’t finished everything we wanted so it’s not published yet, but stay tuned.

Indie Hackers Dublin

So… I would like Dublin to have a more thriving Indie Tech Scene. I think that if the change you want doesn’t happen you can always roll up your sleeves yourself.

So I’m organizing a new community via MeetUp: https://www.meetup.com/indie-hackers-dublin/.

I’m excited about the first meetup that is happening later this week. I have 16 people who said yes and no real experience running meetup communities.

But hey, I’m a capable individual with lots of skills, I’ll figure it out.

Hackathons

So far this year (February), I have done 2 hackathons, one was a casual at my own place, the second one was the Dublin Air Quality Data Hack and I’m signed up for another one: Techstars Startup Weekend Women Dublin.

It is pretty thrilling, even though pretty exhausting.

My experience so far was great, it feels like I’m back in my early twenties.

Not yet started

And there is so much more…

Let’s give you a taste of domains I have bought recently:

myworkoutlist.com
mobilitytutor.com
healthyelbows.com
myweblimits.com <- this is for a new pivot of watchlimits.com

I also still have studytimer.org with nothing on it!

Summary

I think it’s important to know what game you are playing. Sometimes it’s scaling your business and persisting through setbacks. Sometimes it’s fun & exploration.

For now I will focus on having the most fun I can while keeping it sustainable and avoiding grinding myself down. So there will be a lot of things, but progress on any individual thing might be slow. I will also take days off work to compensate for intense hackathon orside project weekends to keep my overall load reasonable.

This choice has its tradeoffs. It’s the opposite of focusing and building momentum on a single project. I will do a lot of crappy things, I won’t have time to finish them off properly.

I expect that this free exploration phase will expand my horizons and will make me better at prototyping and getting things done (I will also have lots of code to copy paste from ;) )!

And if I find something very promising, I might switch back to the exploit mode and focus for real.

I left my Site Reliability job

Sun, 27 Nov 2022 00:00:00 +0000

I left my Site Reliability Engineering job after almost seven years.I didn’t leave Google though, just changed to another role within Software Engineering in the Cloud Infrastructure.

I was somewhat unhappy with my cushy role as a Site Reliability Engineering (SRE) Manager at Google. After quite a lot of deliberation, exploring opportunities and some setbacks (hello, industry layoffs and hiring freezes), I left my SRE job. My last day was October 14th 2022.

I was in SRE for almost 7 years even though I never actively planned SRE to be my career.

How I ended up as an SRE

Let me say this, even though I have worked in Backend Development positions pre-Google, I had interests that would make me a good SRE candidate.

In 2014/2015 I learned about DevOps. I set up all automation and deployment in the startup I worked at the time and I got fascinated by containers and all things cloud infrastructure.

I wrote multiple blog posts about Docker, gave workshops, organized meetups and even had a book deal (seriously! They give book deals to kids like me at the time!). I also was surrounded by various hardware hackers and serious sysadmins in the Warsaw Hackerspace. Hackerspace was my second home, and I would spend 20+ hours per week there hanging out with other hacker friends, building various software and hardware projects.

Even though I was very successful as a Dev in Poland, when Google offered me an internship abroad, I jumped at it. Being 22, it was a great opportunity to get out of my comfort zone. I guess because of my DevOps/Infra interests I got slotted into a SRE internship.

SRE internships are a bit weird, you basically end up being a Dev within a SRE team. It went pretty well and I got excited about the opportunities, the tech was so cool, people were really smart and inspiring. I wanted to try working for Google long term.

And I got it. Even though Dublin was not my first preference I accepted the role and ended up in Dublin as a SRE.I mentioned to the recruiter that I would prefer the SWE position, but somehow “it wasn’t possible”. Well, I was young and didn’t push for what I wanted hard enough. I knew that coming from a SRE internship it was a default path and was just happy about the opportunity in the end.

Inertia and SRE only site

Later on I learned that Google Dublin was an “SRE only site”. All the eng teams were in SRE.

I uprooted my life to move to Dublin, which was disruptive on its own. But later I also added a lot of inertia by:

accumulating significant investments
buying an apartment (1 & 2)
getting married here (to an American - creating a international couple leaving abroad)

All of these would make a move quite a hassle.

Ireland is not the best country in the universe, but it’s pretty damn decent. It has a lot going for it that is aligned with what I value and enjoy.

Well, why not leave Google and get a better suiting job in a different company?

Here are some reasons adding to the inertia:

To interview you need to have enough mental space and practice interviewing.
Google pays very well compared to other companies in Dublin, offers great work life balance and perks (free food, subsidized massages, etc)
After a while my outside of Google options would be SRE or management roles because this is what I was doing currently and had experience in - which if I was to do, I would rather do at Google

It was much easier to just continue and make the most out of it, instead of making a big jump into unknown.

Making the most of it

Even though that I would have preferred to be a SWE instead of SRE, the job gave me an opportunity to work on very interesting engineering problems and codebases with some amazing engineers.

I learned how to do “good enough job” operationally and focused on making impact in a way it was expected from an SRE.

In infrastructure products, most of the difficult, design and code heavy engineering falls onto the dev teams, not the SRE teams.In some ways, SREs shield the dev teams from disruptions and cognitive load to make progress on cranking out features. However, SRE work can still be pretty interesting and I was set on making the most out of it.

I pretty quickly gravitated to leadership roles. Being given trust and autonomy makes any job easier. I quickly realized that many of the realities that we take for granted are just a current setup, which is not the only possible one. The rules and dynamics can be changed. Vision and direction will often stem up from the personality and opinions of the given leader or set of leaders.

I was on a roll, I was productive, I got promoted pretty quickly (L3 -> L4 in 9 months, L4 -> L5 (senior) in 1.5 year, L5 -> L6 (staff) in 3 years).

When things become somewhat unbearable, e.g. I got bored of the type of the work I was doing and types of problems available, I would switch things up. Being promoted helped with this as I was given bigger and bigger opportunities.

I made 3 job changes before I left SRE, Individual Contributor (IC) -> Tech Lead (TL) in my first team, first team to the second team and then in my second team from IC to Manager.

Each time the change energized me and triggered a lot of growth. Changing teams helped a lot. Leaning into leadership empowered me and taught me a lot even though it usually was more stressful than IC roles.

Skill growth

Growing skills is important for building confidence and increasing your value to the team, but also it can be satisfying by itself.

I derive a deep satisfaction from learning. And ideally, I want to keep my personal skill interests and job interests aligned as much as possible.

And I did for a while, learning more about distributed systems, databases, reliability was pretty interesting. (At least for a while, but it’s applicability to my day to day job was limited).

When I decided to switch to management, it unlocked a lot of new learning. It got me out of my comfort zone and forced me to become more mature.

I experienced new organizational dynamics and people dynamics. I had attrition, I hired, I trained, I managed performance, I motivated, I coached.

It pushed me to learn more about myself and other people. I researched books, I read and read, and read. I learned a lot of soft skills and how to be more resilient. But I think I reached a point of being pretty good at my job in that specific environment.

At the same time, my technical learning at the job slowed a lot and the opportunities for learning were not that exciting to me.

Overall, I’ve never been passionate about being very good at troubleshooting, or managing incidents or crises. Probably I’ve been more interested in that than a typical Dev, but comparing to some of the best people I knew, the difference of interest was clear.

In the meantime, I’ve been doing quite a lot of indie hacking and building products from scratch. This also triggered desire to learn, but in many ways it wasn’t aligned with what would help me get ahead at work.

It caused me feelings not having enough time to learn all the things I wanted to learn as my interests weren’t aligned.

Stress, overload, organizational tensions and multi-year projects

I didn’t change jobs because of burnout, but I had moments of burning out before I made the decision to switch.

There was stress, overload and issues. This winter I had some work drama with a poor performer, there were problems with hiring in Dublin and some pushy managers of other teams.On top of it I was training very hard and working on side hustles. It didn’t feel to me as if I was working on my job that hard,but the stress was amplified by me putting myself under other demands and also a fundamental feeling of misalignment.

Side projects made it very clear to me that I need creative outlets, that I want to build my own company, create things from scratch. Meanwhile I had to go to another meeting, write performance reviews and do another oncall shift.

Dev and SRE are supposed to be partners, but often have a tense relationship. In a typical product area, Devs are leading the org shipping new features, making existing infra better and the SRE are exposed to risk and toil coming from these changes. SREs have their own projects, of course, but need to juggle lots of demands and dooming risks.

If you ever heard about circles of concern and influence, a typical SRE TL or manager would have a very wide circle of concern, wide circle of influence and tiny circle of control (due to limited staffing and other commitments). Of course influencing works, but it’s not free and has significant time and cognitive cost. I didn’t enjoy this dynamic too much.

Additionally, I felt stuck with a very long term project. It’s been multiple years in the making, I impacted it a lot and was deep into the weeds. It would keep evolving for many years to come, but wouldn’t be fully done for many years. It was still exciting in many aspects, but the rate of learning has decreased significantly. I felt like it would be pretty hard for me to get untangled from it.

Overall, I realized that this job was making me unhappy and unsatisfied.

I had a lot of decent days, some amazing days, and some “oh crap, I don’t want to do this anymore” days. It’s possible that my unhappy days would just be unhappy and I would find a reason to justify it. Human minds and emotions are not to be fully trusted! It’s also possible that things like my side hustle being a crappy business contributed to some negative emotions.

Finding alignment & imagining the next role

I decided that I will change roles back in May and planned to end my SRE role at the end of September. The team wasn’t in the best place, there were projects I wanted to leave in a good place and I wanted to explore the available options.

I wasn’t exactly sure what I wanted. I knew what I didn’t want - management and SRE.

Pandemic opened more opportunities and people were more open to hiring remotely which was quite exciting.

I dated a bunch of different opportunities. Would project management be too big of a step? Maybe Chrome, it would be so different from SRE. DevRel could give me some useful experience building products intended for developers.

I even bought a book on finding the job to love (which I still haven’t finished!).

I wanted a job that would be better aligned with my long term goals (entrepreneurship and building stuff) or generally a low demand job that would let me focus on those other goals in my free time.

My self imposed deadline was getting closer and I still haven’t found the right job when the internal hiring freeze hit.

Let’s say that it messed up some of my plans. But in the end I ended up with a role that was both a great match for my existing experience and for what I did next. So I guess the long search time paid off?

Leaving good things behind

Even though, the job wasn’t aligned with my long term goals and was making me unhappy, it was hard to change.

As a manager I had my team. I developed pretty deep relationships with people who I managed and also had a great relationship with my own manager and with many other partners across the organization.

The problems with overload also drastically improved over last couple months and in general problems were being addressed.

I was also good at my job and could make solid impact with relatively little effort gliding on my existing knowledge and influence.

It wasn’t terrible. Was I falling a victim of “grass is always greener”? Could the new role be significantly worse?

What next

Directionally, I still want to build a company at some point. I want to grow as a leader. But I also want to grow as an engineer and be a very skilled individual who understands math and science.It’s hard to get all those things at once. For now I will focus on my love for engineering.

Should I put off building a company for another couple years if I wanted to do it since I wanted to be a teenager?Through indie-hacking learned that building a product that people want is much more than just building software.

Could I just do it now? Yes. I think that eventually I would succeed.

Working on indie products made me appreciate employment more

Failing to find significant traction with watchlimits (my indie product) made me appreciate how wonderful paid employment can be.

You get paid a lot, you are expected to be learning, the work life balance is good, you have support network and you work in a space where there is already a lot of traction and your work is almost guaranteed to have impact.

So, since I found a role that is better aligned with what I want long term and has the best what paid employment can offer I will try to make most of it.

I will be working for learning and fun from now on

I am financially in a very good place. I don’t have enough to never have to work again in my life, but I’m not that far off. And I am confident that I will earn money as I enjoy working for the most part.

From now on, my guiding principle for work is that I will try to make my job so good that I would want to work there for free. Main reasons for working being: learning, fun and satisfaction of making positive impact.

I ended up picking up the most intense role available to me, challenging on many levels, but I have confidence that I can pull it off without sacrificing my sanity or health. I’m in my prime years and ready for a challenge.

The future is pretty exciting.

Abandoning the indie hacking dream for now

Fri, 11 Nov 2022 00:00:00 +0000

I’m taking a break from indie-hacking

Will I never write a line of indie code again? Obviously not. I love building, sharing and learning.

But I am going to stop trying to make profitable indie-businesses for the next couple months or a year.

Building things is fun, but marketing a business that has no traction is a grind and a roller coaster. Something that right now I am tired of.

No need for a profitable business

Right now I don’t need to make money with indie businesses. It’s totally viable for me to just enjoy the ride working full time (or maybe later part time) to achieve financial independence. I have saved enough already for a secure retirement at a typical retirement age and if I was willing to downscale my lifestyle, I would probably never have to work again. But I’m in no rush.

I am excited about my full time work, I get paid very well and the environment is very healthy. I feel like in the next year (or more) I have lots of opportunities for personal and professional growth without a need to grind.

And if anything changes, e.g. I get a awful boss - guess what, I can change teams or jobs and be alright.

I know that “would people pay money for this?” is a great test for a business, but it’s not the game I want to be playing now. I might later.

Wanting it all

I like to do a lot of things and “I want it all”.

I want to have an amazing career, be super skilled in my main job, develop my hobbies, know a lot about unrelated but interesting disciplines (e.g. biology, math, physics, psychology), learn foreign languages, and be very fit and healthy (running, climbing, gym, yoga, meditation), creative and have a thriving social life (amazing friendship, marriage and great family bonds). Also I have a thirst for adventure and traveling.

You see where the problem is.

Just having a demanding full time job and a part time side hustle could take most of your physical and mental energy, but I want so much more in life and don’t want to make sacrifices in other areas.

I am expecting my job to become more demanding in the next couple of months and I want to explore more of my hobbies and focus on relationships. So something has to give.

Letting go vs limping along

Leaving things behind can be very freeing. I could maintain things with some effort, but I think that I would always feel behind and guilty of not doing more.

So I am going to cease all marketing efforts that I wouldn’t find enjoyable just on their own and get rid of expectations. I will let watchlimits grow on autopilot and not invest in any new premium features, unless I personally want them :) (I’m a user after all!).

I will also not start working on another indie product straight away.

Did I waste my time?

I put a lot of time and energy into building my side projects like redeal.app (live but unmaintained), invertimo.com (live but unmaintained), watchlimits.com (live and growing slowly, but not profitable).

With each of this projects I learned a lot and the last one watchlimits even earned me some subscription revenue!

Technical learning

I refreshed my grasp on some tech I already knew and learned new one that I didn’t know before. It was very useful as I am working in big tech that generally doesn’t use much open source, but mostly in house stuff. It also enabled me to keep my Dev skills sharp when I was working as a manager or when I just didn’t have many coding projects at my main job.

It generally helped with my future job prospects and overall technical competence and confidence.

Before I started on this journey, ‘can I build a complete product from scratch e2e’ was an open question. There is a lot that goes into it, there are different platforms, UI, backend, billing, etc. I can confidently say now that this is an easy part. I can build everything needed, it won’t be the best ever quality, but it will be complete, solid and quick.

Business learning

Indie hacking taught me a lot about product and business.

All the products that I built, even though not very successful were completely functional and were used by real people and they were rather complex.

It gave me a lot of food for thought in terms of ’time to market’, prioritizing features, target audience and personas, user journeys, etc.

Do I know everything now? Obviously not, but it gave me some reps and lots of practical experience that is complementary to my professional experience.

I think that safely failing at some businesses taught me a lot on what to avoid in the future. So I will be in much better position later on.

Relationship and audience

This journey for sure wasn’t lonely. Okay, at first it kinda was. But then I discovered an amazing community of indie hackers and #buildinpublic.

I also need to give a honorable mention to the Wannabe Entrepreneur community! Tiago is a great community organizer and an indie hacker and made something very special.

Meeting fellow builders made me very happy and excited. So many talented, hard working and kind people out there! Thank you for making my journey much more enjoyable!

What’s next

I will be prioritizing health, relationships, adventures, learning and creative endeavors. I will follow what excites me and energizes me at a given moment.

You might hear less from me on twitter or maybe the topics will just shift. Time will tell, but I’m excited about the new chapter!

My micro-side project trended on HN and got thousands of visits within the first day

Sun, 24 Jul 2022 00:00:00 +0000

My micro-side project Netflix Calculator trended on Hacker News for a couple of hours and:

received 8K+ page visits
brought hundreds of visitors to watchlimits (my main project)
caused many new downloads of watchlimits
got me the first paid customer for watchlimits with a yearly plan
got multiple backlinks and started ranking top 5 in Google within a week - bringing SEO traffic already

OMG! My latest side project - Netflix Calculator - has been on the front page of hacker news for the last 4 hours!

Yes, it's pretty niche! But I am getting some real traffic from this! I honestly expected crickets... and it totally exceeded my expectations!#buildinpublic pic.twitter.com/YNi7ckMxlJ
— Justyna Ilczuk ✨💪 (@attilczuk) July 17, 2022

Here is the traffic for the date (I put it on HN on the 17th July).

Also see a big spike of active users around 17-18th of July on the active users graph:

Inspiration

Getting eyeballs on your project is hard, right? Well, it depends on the project.

Getting traffic to your serious landing page will probably be really hard as it’s not something that people would be excited about and would like to share. But there are some types of projects that can become viral and get a lot of attention.

Those sites are usually simple, fun and produce immediate value for people making them cool and easily shareable. And if executed well, can be a great way to bring qualified traffic to your main project.

If you are an engineer you should consider side-project marketing.

Side-project marketing consists of building a stand-alone product or service to attract potential customers to your business.

One recent example of side project marketing I’ve seen was Books Calculator by Marc Lou. The project was pretty simple and Marc has been getting crazy traffic.

Books Calculator targets similar audience as Marc’s main product - Habits Garden. The side project was fun to build for Marc, got popular easily and was a great source of traffic to Habits Garden.

Was that Marc’s magic or would that be repeatable? I wanted to try it out.

Where did the idea come from?

So why Netflix Calculator in particular?

The idea came from working on the content marketing. I found a post on Quora asking how to check how much time one has spent on Netflix.

I thought that I could describe a couple of different ways, including the Netflix history file and also how to do this with watchlimits, my product. And through the process of thatI found out that all the “calculators” mentioned in other articles were actually broken. Or at least broken for me.

That meant opportunity!

I wrote a blog post and I built a simple calculator online. It wasn’t very pretty, but it worked with my input and I shared the article on reddit. It had some traction.

I decided to double down on the calculator idea and build it similarly to the cute Books Calculator by Marc.

The reasons why I thought it was a good candidate for a side project marketing was that:

it would be quick to build (I had a prototype already)
it refers to the name brand
people are curious and the product delivers value quickly (just input the stats)
I could use Marc’s project as a reference
it would be a nice way to grow my design and frontend skills (I used it to play it with some new technologies as well)

What next?

Even though this launch greatly exceeded my expectations, I’m far from done.

I will continue marketing Netflix Calculator and learning from the experience.

For now I’m not planning to put a lot of development time in Netflix Calculator, but get the basics right - for example making sure it works for all sort of countries. But I don’t dismiss the opportunity of making it much more advanced in the future as I am just scratching the surface with the analytics.

One thing that prevents me from going full out is that I still want to make various improvements to watchlimits not to waste the opportunity all this high traffic is bringing.I timed the first launch of the Netflix Calculator with releasing the first premium feature of watchlimits.

Insights from this first launch helped me to get significant improvements already and there are many more to be done.

Next launches for the calculator:

https://www.reddit.com/r/InternetIsBeautiful/ (next Tuesday)
Productivity related subreddits like r/Productivity or r/GetDisciplined or r/NoSurf (~next Thursday or weekend?)
Product Hunt (early August)

It’s pretty scary, so wish me luck!

Technical learnings from the first quarter of work on watchlimits

Fri, 22 Jul 2022 00:00:00 +0000

You can still use side projects to grow as an engineer without sacrificing the business success of your product.

There are no guarantees that the business will succeed, but I believe that regardless of the business outcome you should try to end up smarter than when you started.

This post is a retrospective of my technical learnings from my first 3 months of working on watchlimits.com (chrome extension to limit excessive video watching). I will separately release a post on my business/marketing learnings as well.

Majority of my professional experience (last 6+ years) is with high performance, distributed storage systems written in C++, reliability and devops. I have some full-stack experience from before then, so I’m not completely new to it, but working on side projects for sure helps me to maintain and develop an engineering skillset relevant to building a wider array of customer facing products.

So here are some things that watchlimits helped me learn/improve at during the first 3 months:

Chrome extension development
Frontend tooling and practices
Design and UX
Mastering ReactJS

Chrome extension development

Before working on watchlimits I had very little knowledge about developing chrome extensions.

And instead of working on something simple I went for a full blown app. I had to learn fast! Chrome documentation was my main source of info, but the documentation is a bit scarce.

watchlimits consists of 4 basic parts:

popup (React App)
content script (Injected React App)
settings page (React App)
background script (just JS app)

I set it up building with webpack with the output files being configure in the manifest file. I set up everything from scratch, before I bought a Chrome Extension Kit. The kit contains code examples of multiple simple apps and an ebook with tips.

Overall the Chrome Extension Kit was good but I bought it too late in the process. I have already figured out most of the things beforehand. I plan to use boilerplate samples for my new projects.

Even though the kit provided a bunch of technical solutions, I still had some important problems left to solve by myself.

The most complex parts I had to develop were around:

storing data & communication between different parts of the extension
updating the extension that runs live scripts (if the background script gets reloaded the old injected content scripts lose connectivity to the background script which was causing issues)
keeping the background script alive with manifest v3 (much harder than it should be)

Frontend tooling - Storybook

My biggest discovery in the frontend tooling was storybook. I heard about it before, but I wasn’t sure how much work would it be to set it up. It turned ou to be very simple and the payoff was amazing.

Storybook stories provide a great way to develop complex presentation components in isolation.

This makes UI development much easier and it can also provide a great alternative to headless integration testing.

Design / UX

I continue to find design to be quite fascinating. I think I am good with basics at this point (contrast, color, spacing, visual hierarchy, etc). But I still have a lot too learn!

Some highlights from this period were:

design systems and design tokens
learning more about common UX patterns like using a modal vs a sub page, where to put buttons on forms or whether or not to use tooltips

I have also found this amazing blog by Adam Silver, full of incredible value articles on UI best practices.

Some of my favorite articles:

More ReactJS

Watchlimits is composed of 3 React Apps!

One of the interesting challenges that I have encountered was mixing useState and periodic closures.

I also made my first NPM package for MUI components integrated with formik, so I can share it between my projects.

The main external libraries I’ve been using were:

MUI (UI components)
nivo (graphs and charts)
formik (easier form handling)
react router v6

And I had to learn the newer version of the react-router as it was quite different from the previous one.

Overall, I have been keeping things simple and almost all of my components have been using functional components and react hooks.

Conclusion

Every project gives us opportunities to learn. Working on indie products we need to balance learning engineering skills with creating value for the customers, but I believe we can do both without too much sacrifice.

What have you been learning with your projects? If you liked this article you can follow me on twitter!

My release checklist for chrome extensions

Sat, 09 Jul 2022 00:00:00 +0000

Many products can be updated very quickly. You spot a bug, you push a fix in next 5 minutes, sadly for chrome extensions it’s not like that.

When you update a chrome extension in the chrome web store you submit a package that then goes through a lengthy review. It can take multiple days and during that time you can’t update a newer version, for example if you find some bugs.

That means that you better deploy solid code otherwise your extension might be broken for multiple days (if the new review takes similar amount of time).

Maybe in some cases of a simpler extensions the review can be simpler, but in the case of my extension watchlimits it’s always been painfully slow!

So how do I make sure that what I release is solid?

I go through a checklist.

Here it is:

Checklist what to do before releasing:

clean up debugging statements
think about main happy and unhappy paths and see if they are covered
update the manifest
build prod version of the extension (I develop in dev mode)
test manually
run tests
run storybook tests
click through storybook stories - all should work and look fine
install extension from unpacked version in chromium (new user experience)
test manually
commit any code changes
upload!
write a changelog

Hard lessons

Cleanups and unhappy paths

The cleanups are pretty self explanatory, no one like chrome extensions to spam their console. But what about thinking about the unhappy paths?

I have the “unhappy paths” on my checklist, because building new features is so exciting it’s really easy to forget about major “unhappy” aspects of the feature!But those matter too! Examples would be - extension is offline or there is no data, etc…

And that can add a significant amount of coding so I do this before any serious manual testing.

Testing & storybook

As your software grows in complexity, it’s easy to miss that something is broken, so testing is key.

It could be manual testing or automated testing. Great automated testing of mostly frontend products is pretty hard. I try to make sure to have good unit tests, but I also rely onstorybook which is incredible for making sure that main subsets of my app are working fine. I also love it for isolated development. I highly recommend trying it out!

Before I release a new version I would look through storybook stories to make sure that everything looks good.

For manual testing there is nothing better than recording a video demo. Because it will force you to go through the whole functionality end to end.

Not only you will make sure that the UX is good, you will easily spot all the bugs that you wouldn’t spot otherwise.

Fresh user installation

If your extension relies on some persistent data, it’s really important to check if the extension will work for new users as well as existing users.

I have made this mistake before! I was developing the feature iteratively and didn’t realize that some functionality required some data to be already persisted and that resulted in a broken experience for new users. Not the best first impression!

Because of that I pack the extension, unzip it in a different folder and run it a separate browser as a fresh installation.

And then I manually test it again…

Changelog?

My app doesn’t have a built in changelog in it, but writing about any big feature release helps me better articulate the value of the feature and also explain how to use it.I can use it for the promotion or as a knowledge base for the users.

Time will tell if it’s a worthwhile effort!

Conclusion

I’m an indie maker, so I don’t have an infrastructure team or time for building a complex CI pipeline or automating all the testing.

What about you? Are you careful with your releases? Do you have any recommendations to improve my checklist?

How I find time for indie hacking while having a full time job

Fri, 03 Jun 2022 00:00:00 +0000

A friend recently asked me:

“How do you find time for all your projects while working full time?”

“I don’t, I just complain about not having enough time instead.”

But more seriously, I am working on two products, while having a demanding full time job and doing a lot of sports (7+ training sessions per week).

Ok, I’m in my late twenties and I don’t have kids, but I do have a social life and a spouse that wants to do stuff together.I still don’t have as much time to work on my projects as I would like to, but I think I make the best out of it considering my priorities.

So, what are my tricks?

Setting clear priorities

“Focus is about saying no” - Steve Jobs

More often than not, you can’t find the time to do the things you want to do, because you are doing too many things already.

I could be working on so many projects, doing so many hobbies and learning so many things!

Yes, I definitely suffer from the shiny object syndrome.

I could be making prioritization decisions every day, but then I would probably end up switching my focus a lot and not being focused in the end. Because of that I like to pick the big “rocks” for the quarter: milestones, commitments, themes.

Why quarterly? For some people months, years or decades might be better, but I really like the quarters as they provide a nice balance between short and long term.

Once I set my priorities, I also set “not a priority” for other stuff. That really helps me with finding time as this cuts out a lot of potential ways of spending my time.

For example one of my priorities for this quarter is growing and getting feedback for watchlimits, not launching some new cool project or e.g. exploring a new tech framework.

Keeping work hours within limits

My job could consume all my available time. I have a great job, but I have a vast scope of responsibility and a never ending todo list. If the more urgent stuff is dealt with, there is always lots of important long term stuff and every week brings some new priorities and interruptions. It would never end and to make it even worse there is stuff happening 24/7. When I finish work, people in USA often start their work days, then people in India. It’s always on.

Burnout is a real thing and it often starts with working too much. I get paid for 8 hours so I will be working 8 hours per day and if I feel too tired before 8 hours are up, I will cut the day short.

A person can rationalize staying up late, working over weekends, etc. “No one else can do it”, “I have this important deadline”, but really what’s the worst thing that is going to happen if you don’t finish this thing?

Maybe you will disappoint someone. They will likely understand. Also, you will have to learn how to better plan to be in a situation when you are late less often.

Anticipation & planning help with avoiding all sorts of “OMG, we have to stay late for X” problems.

But a different likely outcome is that people will respect you for having boundaries, and you will be a healthy example for other team mates.

Cutting out the time sinks

Everyone has their own time sinks. For me it used to be video games and watching videos like netflix or youtube.

It’s engaging, it’s enjoyable, but then most of your free time is gone!

Obviously, it’s not always bad, you might decide that those things are really important to you and keep them as your priority. But you might also realize that they prevent you from doing the things you really want to do. Like working on the product on the side or learning skills to change jobs.

These days there are so many great shows to watch on netflix and it’s so easy to binge content. Similarly for youtube, the interesting content is never ending. The algorithm will try to keep you glued to the screen.

What I use day to day is:

Disabling apps on android (gmail on corporate phone, sometimes chrome on my personal phone)
Blocksite - I use the free version
watchlimits extension (that I built myself) - that one specifically help me not to fall into “one more netflix episode trap”.

I also wrote an article with a 3 step framework to find additional time by reexamining your digital habits.

If you cut out your major time sinks out of your life you will find yourself with lots of unoccupied time. For some it might be super boring, for me it feels luxurious.

Summary

There is no magic here, but those 3 simple tricks work well for me:

Setting my priorities clear
Keeping my work hours capped
Avoiding excessive time sinks

It’s still hard to work on the products part time and I have more or less productive weeks, but I try to make regular progress.

Feel free to share what works for you! You can follow me on twitter!

How to make a product on gumroad and earn some money (#secretunasaladrecipe)

Sat, 09 Apr 2022 00:00:00 +0000

If you’ve been in the Indie Hackers circles for a while you probably heard about people like dvassallo@ who made hundreds of thousands of dollars selling simple digital products on gumroad.

Make a thing, put it on gumroad, earn money, could it all be that simple?

I have heard many good things about the platform and I wanted to try it out with something straighforward. I didn’t feel like I had some super valuable secret I could share that I could make into a killer course, but…

I had the incredible secret tuna salad recipe (motivations) though and I decided tomake it into a gumroad product and I even made some money with it.

If you think I should be earning more money from this, here is your link to buy it :D.

In this article I will explain what went into making “the secret tuna salad recipe”, the digital product.

Ideation and planning

What should be included in the product? I had a couple of ideas.

The core of the recipe should be in a PDF as it can be good to present lists, steps and explain the details

I was thinking about making a video but video production of recipes is a pretty big undertaking and since I’m not a pro youtuber the video production quality would likely be poor (my husband also started laundry so that eliminated the possibility of shooting the video that day).

I was also thinking about including some bonus content with tuna or fish jokes, but I decided that I’m not that funny and I couldn’t find enough funny jokes…

At the end you have to play to your strengths. Including something of poorer quality might dilute the value of the product and leave the customer disappointed.

I used notion for planning and brainstorming ideas. I also wrote there my shopping list, ingredient lists and all the steps of the recipe.

Once I had all the ingredients ready, I decided to take detailed photos of the required equipment and ingredients and later document each step of making the recipe.

Gear and tools

I have a tripod I could use, but it wasn’t necessary as I wasn’t really taking pictures of myself, but of things.

I took handheld pictures with the Pixel 6 (maybe one or two were taken by my husband when I needed an extra hand).

I am a Google Photos user and my photos were quickly available on my computer.

How to make a nice PDF with Google slides

I use linux and I don’t have Microsoft Office installed, but I know that it’s pretty easy to make a PDF out of Google Slides and it was my tool of choice.

For the PDF to look good, I wanted to use a consistent visual theme. I decided to make my own visual theme out of blue with a small touch of yellow and Nunito font.

The screenshot below shows how to make your custom theme in Google Slides (Slide > Edit theme):

Once you have your slides ready, you can easily convert them to PDF with File > Download > PDF Document.

Visuals

I wanted to add some nice visuals to make the recipe more interesting and I searched around for some good visuals. One obvious source of expressive images is of course emojis, but I wanted something more distinct.

Also, I wanted it quickly and ideally for free. I could draw something myself, but that wouldn’t be quick and I’m pretty bad at digital drawings as well, so that option was out of the question.

My usual source of visuals is a fantastic pexels.com, but I didn’t see anything that would match my artistic vision (and the type of tuna I was using).

I had good luck as Wikipedia had lots of great tuna illustrations in the public domain! Score!This is how the mister helpful tuna was born:

Setting up the gumroad page

Once I had my content, my visuals and a generated PDF it was time to put the product on Gumroad.

I expected it to be much harder, but it’s surprisingly simple.

You start with the name and a price:

And then you can customize what will be shown to the user and what files they get when they buy the product.

Probably the longest part of the process for me was setting up the images, especially because the thumbnail had to be of very specific size.

For the selling copy, I decided to keep it short and simple!

If you have some suggestions, I would be happy to hear them.

The rest is history marketing!

Once you hit publish, it’s out there and people can buy it.And some did!

However, as with any product you can’t just create it and wait for people to come to you, you need to market it.And that’s a completely different story :).

From the gumroad side, it has some useful tools for marketers such as analytics (to track where people came from and which sources convert the best) as well as it collects emails of your customers and that can be used for getting feedback on the product, sending updates or upselling.

If you enjoyed this post, you can follow me on twitter or support me by getting the tuna salad recipe or one of my other gumroad projects. Keep on hacking!

Sometimes stupid product ideas are the best - motivation behind the secret tuna salad recipe

Thu, 17 Mar 2022 00:00:00 +0000

I created a digital product out of my family’s favorite tuna salad recipe and in this article I will tell you why I thought it was a good idea.

I’m a software engineer by trade and I have spent thousands of hours building software products that don’t really earn me money (yet).

Entrepreneurship is risky, that’s for sure, but often indie businesses fail because founders lack marketing skills and fear to put themselves out there until it’s pretty late. And then it turns out that they have been building a product that no one wants.

My inspiration

During my last vacation I got a bit fixated on a weird idea - making a digital product out of my family’s favorite salad recipe.

What got me excited was that:

it’s a great tasting, healthy salad and it’s very easy to make
it could be fun to make a big deal out of it and sell the recipe as a digital product on gumroad
in a process I could learn: marketing, SEO, ads… and popularize a great salad so what would be there to lose apart from my time?

The #secrettunasaladrecipe operation is on!
I will make a digital product out of it and sell it on gumroad!

Today's progress:
- bought a domain https://t.co/Fs1k2KfqR2
- keyword research
- failed to make the salad due to a lack of a key ingredient (not tuna)#buildinpublic pic.twitter.com/15LnxFWOSF
— Justyna Ilczuk ✨💪 (@attilczuk) March 8, 2022

If something is good, making money from it is just a matter of solid marketing, right?

Are you mad, this can’t be a good business idea, right?

People are used to seeing recipes online for free. It doesn’t make sense to put in a lot of effort if people can access countless good recipes without paying. The recipes are too common.

However, you could say the same about books, there are countless books out there, many of them accessible for free at your local library. But sometimes you want a specific book, maybe because it was marketed to you recently or because others recommended it. It’s not a huge cost and you just buy it. I think in this case it can be similar, as long as it prices accessibly, I think people would be ok to pay for it.

The other argument against this project, is that no one will want it. Who is interested in tuna salad anyway? Well, the basic research shows otherwise:

There are between 100k to 1 Million monthly google searches for the “tuna salad” alone. And there more other keywords that look interesting too. The competition seems low. I smell opportunity!

What makes this idea a good business?

The underlying product is really good and the founder (me) believes in it. There is a good reason why this recipe is so popular among my extended family. Because of that I am happy to put great effort behind it even if it’s a bit silly.

Another great reason is that there are no ongoing costs and no costs of physical goods. Once the recipe is packaged into a product it can be sold at no additional cost! I plan to improve the product by giving location based tips (due to accessibility of the ingredients), but apart from that there is no need to ever change it as the recipe itself is pretty good.

This product is also a great way to learn marketing. There is no better way of learning than by actually doing and this product will provide countless opportunities for marketing and in many ways it’s better for learning marketing than if it was a serious SaaS startup.

Learning business on digital products vs SaaS

Building a successful SaaS is really hard and there is a huge start up cost of building a good product before being able to sell it and it being good enough to be really used by customers.

It’s easy to get bogged down with all the complexity. It’s better to start with something simpler.

Here’s a quote from Rob Walling blog post on stairstep approach:

In my experience, the biggest pitfall that trips up first-time product people is trying to create something too complex.The strategy that seems to give people the best chance of success is creating a simple product, with a simple marketing plan – one that only requires a single traffic channel.

I’m not the only one thinking this way!

Still sounds fishy?

In the end I am doing this because it’s original and fun.

The idea brings a smile to my face, every time I think about it. And I hope other people will appreciate the weirdness of it too :).

So yeah, now I sell a secret tuna salad recipe online!Wanna buy?

Decimals in python, Django and your DB

Sun, 13 Feb 2022 08:55:59 +0000

In programming we use numbers a lot. For counting integers are quite nice, but for many use cases we need “real” numbers which apart from integers also include numbers like 1/3, $ \sqrt{2} $, etc. Typically these are approximated with floating point numbers based on base 2. Traditionally we call them floats .

floats are great for many use cases, but they have their issues.

Why use decimals?

One big issue is that floats can’t express many of real numbers that are commonly used by humans in daily life without some approximation.

Unfortunately, most decimal fractions cannot be represented exactly as binary fractions. A consequence is that, in general, the decimal floating-point numbers you enter are only approximated by the binary floating-point numbers actually stored in the machine.

Here is an example:

>>> 1 + 1 + 1 == 3True>>> 0.1 + 0.1 + 0.1 == 0.3False

That applies to other types of fractions as well, like 2/7 or 1/3 which also can’t be represented accurately.

Some numbers can’t be accurately expressed as fractions like square root of $ \sqrt{2} $ or $\pi$, but for the numbers that can (rational numbers), python has a solution!

fractions module - will help you deal with any type of fractional numbers
decimal module - is specialized with decimal numbers like 0.12 or 99999.999877 which is very useful in business and finance as this is what humans use for money.

This article will focus on the use cases that require decimal arithmetic (e.g. anything related to money) and therefore the use of decimals .

Decimals in action

Python’s standard library has a decimal module, similarlyit’s very well supported in Django and majority of the SQL databases.

Decimals in python

To use decimal numbers in python you need to import the decimal module and create your decimals with decimal.Decimal

>>> import decimal>>> decimal.Decimal(2) * decimal.Decimal("0.75")Decimal('1.50')

When you create a decimal number, you should do it from either an integer or a string (or a specially formatted tuple, but that’s more complex). Creating a decimal from a float also works, but there are problems associated with that and I will explain it in the pitfalls section.

Python has a feature called operator overloading and thanks to that you can do normal arithmetic on your decimal numbers without a need to call methods, just with + , * etc.

You decimal.Decimal instances also have various methods, the most useful of which isquantize.

>>> Decimal('7.325').quantize(Decimal('.01'), rounding=ROUND_DOWN)Decimal('7.32')

Decimal operations in python are configurable. You can easily change how many digits after full numbers you want and what will happen if you need some rounding or if your arithmetic is imprecise.

Luckily you don’t have to configure this per each decimal. Decimal instance, that would be very tedious! Python’s decimal module has a concept of a context where all the possible settingslive and can be overridden.

There is a default global context that you can access and change, or you can use a local context that will be restored after you exit a code block.

Here are two examples of making the same change to the context, global one:

from decimal import getcontextgetcontext().prec = 42Decimal(2).sqrt()

Local one:

from decimal import localcontextwith localcontext() as ctx:    ctx.prec = 42   # Perform a high precision calculation    s = calculate_something()

Contexts are pretty neat and they can allow you to configure a lot of things, like:

how to round
how much precision to use

decimal module also supports Signals which represent conditions that arise during computations, like “Inexact” or “Overflow”.These can be configured to raise exceptions through the decimal context or just simply accessed and cleared.

Decimals in Django

Django supports decimals through a special models.DecimalField.

You can configure the amount of digits and decimal places. The decimal places are included in the digits.Example model with a precise decimal:

class PriceHistory(models.Model):    asset = models.ForeignKey(Asset, on_delete=models.CASCADE)    value = models.DecimalField(max_digits=20, decimal_places=10)    date = models.DateField()

Similarly, it’s supported in Django forms and django rest framework.

class PositionSerializer(serializers.ModelSerializer[Position]):    asset = AssetSerializer()    # Latest price is not defined on the model directly.    latest_price = serializers.DecimalField(max_digits=20, decimal_places=10)    class Meta:        model = Position        fields = [            "id",            "account",            "asset",            "quantity",            "latest_price",            "latest_price_date",            "latest_exchange_rate",            "realized_gain",            "cost_basis",        ]

There is no need to specify max_digits and decimal_places for a field that is both in the model and in the serializer.

Decimals in the SQL database

Decimals are well supported by majority of SQL databases (PostgreSQL and MySQL) as it’s part of the SQL standard.

Note: SQLite doesn’t have support for actual decimals. So things might work in unexpected ways.

The implementation between Databases might vary, for example PostgreSQL supports many more significant digits than MySQL (by multiple orders of magnitude).

The decimal SQL standard requires precision and scale to be configured when the decimal column is added. The meaning of those parameters is equivalent to Django’s max_digits and decimal_places.

So what can this precision cost us? More digits incurs storage overhead.

From PostgreSQL documentation:

Numeric values are physically stored without any extra leading or trailing zeroes. Thus, the declared precision and scale of a column are maximums, not fixed allocations. (In this sense the numeric type is more akin to varchar(***n***) than to char(***n***) .) The actual storage requirement is two bytes for each group of four decimal digits, plus three to eight bytes overhead.

So my Django decimal with 20 max digits and 10 decimal places will require 2 * 5 + 8 (worst case overhead), 18 bytes per number!

A double precision number would be 8 bytes.

That difference can add up pretty quickly.

Downsides and pitfalls

Now that you are convinced that decimals are often necessary and you have learned how they work, let’s talk about how you canshoot yourself in your foot using them.

Mixing floats and decimals

Decimal-float operations are not supported for a good reason.

>> decimal.Decimal("0.3") + 0.8TypeError                                 Traceback (most recent call last)<ipython-input-6-92fb256311c2> in <module>----> 1 decimal.Decimal("0.3") + 0.8TypeError: unsupported operand type(s) for +: 'decimal.Decimal' and 'float'

In the case above 0.8 is not exactly decimal.Decimal("0.8") it would have to be convertedto a decimal and that could result in some imprecision. If you hit this type of errors, it meansthat some of your numbers are not decimals and they can make the result of your computationsnot exact in the end.

So what to do? Just convert the float to decimal, by using decimal.Decimal(my_float_number)? Well, no…

If you create decimals from floats you are loosing accuracy

This another pitfall, as every time you create decimals from floats, your number get imprecise.

>>> from decimal import Decimal>>> Decimal("0.1") + Decimal("0.1") + Decimal("0.1") == Decimal("0.3")True>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) == Decimal(0.3)False

Instead, keep the numbers in your system as strings, integers or decimals without ever converting them to floats.

TIP: use type annotations to annotate all decimals.

Default context vs local context

I would generally advise against modifying a global context in decimals in your code, unless you have a simple program and you clearly do it in the beginning of its lifetime.

If you modify it at random points, you might end up with unexpected values in your context if you forget to clean them up!

A much better solution is to always keep it local with localcontext.

from decimal import localcontextwith localcontext() as ctx:    ctx.prec = 42   # Perform a high precision calculation    s = calculate_something()

You might even want to declare your custom decorator for that.

Django pitfall: forms/serializers out of sync with model field definition

It’s not only that you don’t need to specify max_digits and decimal_places on a matching field between model and serializer, you shouldn’t.

Why? Any time they get out of sync it can lead to problems, one type if it’s bigger on a serializer and one type if it’s smaller than on the model.

When serializer allows for more digits then user will eventually put more digits there. Their input will pass the validation but then fail at the database level raising an exception and failing to the user.

When serializer allows for less digits than the model field, then e.g. when the value is edited by the user it might get truncated even if user hasn’t really made any changes to the value. This can happen if the user changed other values, but the API is passing all the values again.

Both errors to the users and unexpected losses of precisions are pretty bad, so pay attention to this as it might be hard to catch in tests.

Django pitfall: precision of decimals and duplicate prevention

I’m a big fan of using get_or_create in Django to prevent duplicates when the value is created with the same arguments.

But the arguments won’t be the same if the decimal’s precision differs!

So be careful with that as decimal module precision defaults to 28 places and that is very likely not what you have configured on you model.

To deal with that you can override your context to use required precision when performing arithmetic, or even better you can use the quantize method on the value.

References & Resources

I hope you enjoyed this long post! Please follow me on twitter and share it if you found it useful!

Additional resources:

Invertimo Binance Crypto Integration

Wed, 02 Feb 2022 13:15:59 +0000

You can now import your binance Crypto related transactions to invertimo.com (open source investment tracking app).

Why track crypto transactions in invertimo?

Invertimo helps you track all transactions related to your investments as well as dividends and received income in one place.Crypto is becoming more and more mainstream and more investors have part of their portfolio in crypto assets.

Selling crypto is a taxable event in most countries, swapping tokens is usually treated similarly.

Because of that you need to be on top of your transactions and know what was your costbasis (total amount paid) of the tokens you sold. You need to make sure you track it in the correct currency,converting USD gain to your local currency is not the way to do it. Additionally, order of selling matters as well,even if you have the same token in multiple wallets or exchanges! Do yourself a favor and keep a solid record of your transactions!

Crypto can be an income producing asset and you need to track this income too. The income is usually produced by Savings products (similar to bank savings accounts) or through staking (helping secure the blockchain). Tracking the income can be especially challenging and you can read more about it in a section later on.

Why binance?

Binance is one of the biggest centralized exchange through which many people buy crypto.

It’s also the exchange I’ve been using personally, so I prioritized building the integration as I could be my own test rabbit.

Crypto Income & problem with daily interest

When you receive interest from your tokens two things happen:

you receive income of the value of the tokens you get
you receive the tokens with a cost basis of the income you just received

Let’s see it in an example, let’s say you stake a token DOT. It has a high interest rate and produces income daily.

On Monday you receive 0.02 DOT. The price per token is ~20 USD on that day. You are based in Europe and taxed in euros.

You received 0.2 USD of income! 1 USD is 0.88 EUR at that day, so you had an income of 0.352 EUR. You don’t see it in your account balance, but instead:
you have additional 0.02 DOT that if you sold next week you would have to pay capital gains on. Their cost basis was 0.352 EUR.

So that was for Monday, a couple days later the price might be completely different as crypto assets are quite volatile.

And guess what, currency exchange rates fluctuate a lot too!

And if you own multiple different tokens that have such daily interest, the amount of interest events and transactions quickly adds up!It can reach thousands in no time.

It happened to me in a couple of months.

Data sources in invertimo

To get the crypto prices and currency exchange rates, you need a reliable data source. Invertimo gets its market data fromeodhistoricaldata.

If you trade tokens for fiat currency, the app uses how much fiat you paid as a price used in that specific transaction.

How it works

CSV import

Once you export your binance transactions (instructions here), youcan upload a .csv file to invertimo.

The binance parser under the hood will create all necessary transaction records and events.

Transparency

You can see exactly what happened in the given import. The import can be successful, partially successful or fail completely (unlikely!).

Any transaction or account event that was created as a result of an import has an associated import record that stores information used for the creation of that transaction or event so you can inspect it.

Deduplication

One of the design principles of invertimo.com is that it prevents duplicate records. So there is no need to worry that something will be recordedmultiple times even if you upload history files with overlapping transactions.

We will store in the database that the record was imported, but deduplicated.

Current limitations

Only SPOT account transactions are supported and only the following binance operations:

BINANCE_SUPPORTED_OPERATIONS = [    "POS savings interest",    "Savings Interest",    "ETH 2.0 Staking Rewards",    "POS savings redemption",    "POS savings purchase",    "Savings Principal redemption",    "Savings purchase",    "Deposit",    "Transaction Related",    "ETH 2.0 Staking",    "Withdrawal",]

In other words:

transferring money in and out of binance
buying or selling crypto
staking interest
savings interest

I’m not sure how exhaustive this set is, but if you have more I should support, I would be happy to add it.

Alternatives

If you want a solution just for crypto taxes, there exist some solid alternatives!In particular I have been using koinly.io so far!

The problem is that it charges a lot if you have crypto income:

With my couple months of transactions, I would have to pay them almost 200 USD per year.

Invertimo is not perfect (under active development!), but it is open source and even though it won’t compute the taxes for you,it will help you track the transactions and income and that is more than a half of the battle.

What next?

The crypto support in invertimo is far from done. I am planning more UX improvements and better support for crypto transactions outside of the binance integration.

The journey is never over! If you enjoy this article, please follow me on twitter.

Browser caching with Django & Webpack

Sat, 08 Jan 2022 10:45:59 +0000

What is browser caching and why is it useful?

Fetching stuff from the internet can be a lot of work and take a long time. What if your browsercould save itself all this work and return you the result semi-instantly?

It just needs to save a file locally and return it to you next time you want it. We call it browser HTTP caching.Passive operations like “getting” the page will usually be cached.

This is great as long as the file at this address doesn’t change. But the thing is that it often does, especially if your site is under active development.

When things are just not working the way they are supposed to or styles are off, it’s very likelythat the problem is unintended caching.

Crude approaches

Two popular crude approaches are:

disabling caching
manually managing the filenames after edit

Disable all caching!

One simple, but crude approach is to disable caching.

If caching is the source of my problem, let’s disable it!

In your browser

This is a good approach if you are not sure what is your problem and want to quicklyverify if caching is to be blamed.

Modern browsers have an option to disable cache in the developer tools:

After you disable caching in the options, reload the page to get new content.

This is one of the “works on my machine” 🤦 types of solutions and you can’t expectyour users to clear or disable their cache just because your app doesn’t handle caching well.

We need something better!

On the server side

The good news is that the browser will do what you tell it to do. You can set certain response headers ( Cache-Control ) tocontrol the browser cache behavior when you send back responses from your server.

If you use nginx (a web proxy server often used with Django), you can disable caching there.

If you use the Django’s static convention, nginx config that disable caching would be:

server {    location /static {        alias /var/www/mysite.com/static;        # kill cache        add_header Last-Modified $date_gmt;        add_header Cache-Control 'no-store, no-cache';        if_modified_since off;        expires off;        etag off;    }    ...}

This approach works, but it has a disadvantage. Files that don’t change and could be cached, don’t get cached and you are missing outon site speed ups.

Change the filename/import every time you edit

If files get cached if the URL is the same, why not change the URL if the file changes?

Renaming the file every time you edit would betedious and in general, we have version control so we don’t have to create a new file every time we make changes.

But there exist a related technique of changing the URL by adding “?version=XXX” a versioning query parameter.If you are just serving static files, without any custom logic, the additional parameters are ignored by your server,but the browser will treat it as a new URL.

So instead of:

<script src="/static/my_script.js"></script><link rel="stylesheet" href="/static/base.css">

you would write:

<script src="/static/my_script.js?version=123"></script><link rel="stylesheet" href="/static/base.css?version=123">

and in case of django with the static template tag:

Instead of:

<link rel="stylesheet" href="{% static 'base.css' %}">

you would write:

<link rel="stylesheet" href="{% static 'base.css' %}?version=123">

With renaming you get a finer control than just disabling cache, because the things that can be cached stay cached.But it’s still not a great solution.

Using a version parameter in the URL is a better approach than just renaming the filename every time you make the edits yourself, but it’s still manual and error prone.It’s easy to forget to do it. Why not automate it?

Content hashes in filenames

One good way of updating the file name any time the file changes is by usinghashes of the content as part of a file name.

Doing this all the time, would be a bit expensive, so it’s usually done during a build process.

If you are using Django together with React (or Vue, or…) in a way that Django serves page that thenloads a JavaScript bundle built with webpack then you build has two phases:

webpack build to build your JS assets
Django collectstatic to do any preprocessing and move files to the static location.

Typically the way it’s set up is that your JS code is in some directory, e.g. called assets and when your JavaScript bundle is compiled thebundle is put into /static directory, the default location for Django’s JS and CSS files.

You can read about how to set up Django with JS like this here.

Let Django do it for you

Turns out Django has a way to automatically add hashes to your static files any time the content changes.

Just update your staticfiles storage implementation to ManifestStaticFilesStorage in your Django settings.py file.ManifestStaticFilesStorage computes the hashes based on file’s content, adds the hashes to the file names and then updates all the import paths that use static tag appropriately.

Nice!

Well, that works, unless you use code splitting and hashing with webpack. Unless your JS app is really small, you shouldn’t have it all in just one file. This is where the trouble starts with using the Django manifest solution.

Let webpack do it for you

If you are still reading, you are probably curious how manage things on the JavaScript side if your bundle consists of many files.

Well, webpack has support for caching by generating file names with content hashes too.

But then once you have a bundle with a hash in its name, how to integrate it to work with Django.

Google search suggests to use django-webpack-loader

This tutorial talks in detail how to set it up.

This solution requires a custom plugin on the JavaScript side and a custom package and app on the Django side.

Quite a setup to just load a bunch of correct files. I’m always skeptical when it comes to taking on new dependencies.

The django-webpack-loader solution is not native to webpack and it broke in 2018 when webpackchanged some of its defaults. django-webpack-loader solution is rather fragile, it depends on reading webpack stats that are a byproduct of a webpack build (if configured).

While reading the Webpack caching documentation I came up with a simpler solution that doesn’t require any new dependencies.

Webpack & Django integration without any additional packages

Loading a bundle with a hash in a name is a common problem, not specific to Django. And webpack now has a native solution tosolve it. It does that by generating a html file that will load your bundle.

A html file, just like the Django templates…

Can webpack generate a Django template? Of course it can! If you don’t need anything Django specific then you don’t even have tocustomize it, but if you do, it’s easy to specify a “template” file for webpack to use.

Here is how it can be set up:

const path = require('path');const HtmlWebpackPlugin = require('html-webpack-plugin');module.exports = {    plugins: [        new HtmlWebpackPlugin({            // This is the template webpack will use to generate html file.            template: 'templates/index.tmpl.html',            // This is where the generated file will end up (relative to the `static` directory).            filename: '../templates/index.webpack.html',        })    ],    entry: './assets/index.js', // Path to our input file.    output: {        filename: '[name].[contenthash].index-bundle.js', // Output bundle file name.        path: path.resolve(__dirname, './static'), // Path to our Django `static` directory.    },    // Code Splitting and bundle optimizations.    optimization: {        usedExports: true,        moduleIds: 'deterministic',        runtimeChunk: 'single',        splitChunks: {            cacheGroups: {                vendor: {                    test: /[\\/]node_modules[\\/]/,                    name: 'vendors',                    chunks: 'all',                },            },        },    },    // Other webpack configuration...};

Contents of templates/index.tmpl.html

{% load static %}<!DOCTYPE html><html lang="en"><head>    <title>My page</title>    <!-- Webpack imports with auto generated bundle names that avoid caching if data changes. -->    <!-- My other things... -->    {% block more_head %}    {% endblock %}</head><body>    {% block content %}    {% endblock %}</body></html>

Now when webpack builds your JavaScript assets it will place its outputs in the /static directory and also createan additional file in your templates directory. In the case of above example it will generate/templates/index.webpack.html that will load relevant bundle entry points (one or more).

This solution might look a bit complicated, but it’s actually pretty straightforward, it just has two build steps and webpackcreates files in two directories, in /static directory and in /templates .

Combine Django and webpack file caching management

You might have figured out that if both webpack and Django generate content hashes, you might end up with a lot of hashes and thatmight break lazy module imports in your JavaScript app.

The easiest way to circumvent that is to either stop using the Django solution, or to make it ignore the webpack generated files.

Probably the simplest way is to change the way you call collectstatic . As in the webpack configuration in an example aboveall the files generated by webpack have index-bundle.js suffix:

python manage.py collectstatic --ignore *.index-bundle.js

Conclusion & resources

So now we have multiple solutions for always loading the fresh version of the file when it changes.And all the files that don’t change can now be cached for a very long time speeding up our site ✨.

I hope you enjoyed this article, please follow me on twitter where I talk more about tech, python, Django and building software!

If you are hungry for more, here is a list of resources I refer to in this article you can read as well:

Building Invertimo in the open (as open source)

Sat, 08 Jan 2022 10:45:59 +0000

As far as building in public goes, it’s hard to go more public than making all the code public too! I am building the Invertimo (investment bookkeeping and tracking software) completely in the open. It’s a complex web app written in python and JavaScript.

I submitted 200+ commits over last couple months.

Github repo.

I got the rights

I work at a bit tech company that by default owns everything that I build. There is a process though to ask for permissions to retain the copyrights as a creator. If I have the rights, my company doesn’t care if it’s open source or not, it might as well be. There was no conflict of interest here and it’s not related to my day job, so I was free to own the rights for this specific app.

What if someone stills my idea or copies my code?

Well, good luck with that. The app doesn’t do any income, so why would anyone?

My code at times is not that polished, what if people judge me?

I have pretty high standards when it comes to development and sometimes, the code I submit on personal projects is not my best work (but it’s still pretty solid overall, I think). Well, I don’t get paid for writing this and I have limited time to work on the project on the weekends, so I will be cutting some corners. As long as I still have fun writing this, things are good.

On the other hand, it’s cool to share your work like this. It proves that you can build complex things that work and not just tweak things that other have built.

Show casing real applications

There aren’t many examples of complete apps that are publicly accessible like this. It’s a good showcase of solving real problems.

Source code of complete working applications was a great resource to me when I worked in a software house a couple years back. I could easily see how a specific problem was solved and how things were architected. It saved a lot of time.

To give you some taste of what I have built:

Portfolio overview page

Onboarding

Realized gains

And much more… Play with it if you want at invertimo.com.

Technology used

I use pretty standard (boring) technology for invertimo. Here are the details:

Backend:

python 3.8
django
PostgreSQL

Frontend:

JavaScript & CSS (I haven’t jumped on the TypeScript bandwagon yet!)
React
webpack - package management
material ui - complex frontend components

Infrastructure:

docker - all app components run in containers and are be built in a reproducible way
nginx - web server reverse proxy
Digital Ocean - simple and friendly cloud provider with very clear billing

This set of technologies serves me pretty well for building complex web apps.I believe with some tweaks it could also scale very well.

Major inspirations for setup & architecture

It is hard to create things entirely from scratch and I didn’t do that. I found the resources below especially helpful:

https://www.obeythetestinggoat.com/ - a fantastic open resource about how to build and test a django application end to end
https://www.saaspegasus.com/guides/modern-javascript-for-django-developers/ - this was especially helpful for putting Django and JS frontend together.

If you build modern django apps, those resources are golden.

What next

I don’t have much commercial plans with this app, but building it is quite fun and I’m solving my own problems. There is still a bunch of features I could use that I haven’t built yet.

In Q1 2022 I plan to build support for crypto related tracking (e.g. income from staking or crypto savings) and binance (popular centralized exchange) integration. It is going to be a lot of work, I’m sure.

If you enjoy this post and what I am doing, please follow me on twitter. Also feel free to say hi!

I prefer boring technologies

Thu, 30 Dec 2021 10:45:59 +0000

When I start new projects I try to build them with boring technologies that I used before and I try to limit picking up new libraries, tools and technologies to a minimum.

This is because I want to build actual products and not to do projects for the sake of learning exciting technology.

I chose boring technologies because:

they are proven & dependable
I can focus on building a product
challenges specific to a product keep it interesting and I don’t need shiny tech novelty

If you want to build cool things, you should consider doing the same.

Boring == proven & dependable

Often the reason why something is boring is that it’s been around for a while and it’s popular.

Those are good things! This means that many bugs have already been found and fixed. This means that many people hit various issues with this technology and lived to tell their tale.

On the other hand many new shiny things have lots of unknown issues. You might be the first person hitting a particular problem and that can happen to you every month.

Another issue with shiny tech is that it might not be here anymore after a couple years if they don’t get wide adoption. Boring technologies on the other hand have communities of maintainers and many companies behind them.

Boring == familiar → can focus on building a product

What is boring is often familiar which means that I am more fluent using it.

Instead of focusing on how I can implement an API, I can focus on what I want within the API. The implementation is straightforward.

I really believe that you have a limited number of “innovation tokens” while working on something new. You can spend them innovating on the product or on learning a novel technology.

However unless the technology itself is a distinguishing factor for your business or product, you should minimize using your tokens on the novel tech. Your users won’t care if you are using Vue or Svelte or React as long as your app is fast and easy to use and solves their problem.

Technology stack vs technology alternatives

Is the framework the fastest and the coolest? Probably not, it’s possible that there are alternatives that are slightly better.

I want to focus on learning and mastering a set of complementary and not alternative technologies, because this makes me more effective developing things full stack myself.Alternative technologies would be like React and Angular or Python and Ruby. Instead, I develop an expertise in a stack of complementary technologies, like JavaScript and SQL or python and dev ops.

Having many alternative solutions deployed increases your complexity significantly. The technology combinations in your stack grow very fast.So keep your tech life simple and focus on the product instead.

Boring technology - but an interesting project?

Won’t you get bored if you are only using proven technologies or keep using the same libraries over and over?

This is a risk to some extent, especially if you have lots of boilerplate. Many products require the same things: users, authentication, payments, email. But after you get the basics out of the way, then what is left is specific to the problem you are going to solve and that is different with each product.

And there are so many unique challenges to each product, the data modeling, API design, UX, design and so on!

Building end to end products is exciting!

Summary

To sum up:

boring technologies are more dependable
focusing on couple of solutions creating a consistent stack helps me master them and makes my life simpler
developing products is often more interesting than dabbling in technology for technology’s sake

If you want to read more about boring technologies, I found these slides pretty helpful.

If like stuff that I write, please follow me on twitter where I post about technology and my Indie Hacker journey.

Learning can be a waste of time

Sat, 13 Nov 2021 10:45:59 +0000

Learning and studying isn’t universally a good thing.

I found that saying “sometimes spending some time studying and learning can be a waste of time” is very contentious. Especially in a quickly evolving tech environment.

If you stop learning and become rigid in your ways, you become obsolete. If you stop learning you miss the opportunity to get more efficient and be able to do more in a shorter time.

But, we shouldn’t just glorify the activity, because it’s essential when executed well. Not all studying or even learning is a good use of your time.

Similarities with finance

Let’s compare learning to financial investing. Investing is generally perceived as good for you. But is spending money buying “investments” always a good thing? What if your “investment” has zero return of investment and is just a waste of resources? All because you didn’t do enough research and made poor capital allocation?

Or when you do too many investments and get overstretched. Then you can’t put enough attention to the project and then it fails.

Or if you got excited by something shiny (hello, crypto-dog-coins) and lost lots of money because of speculation? Maybe in that case it would be better not to “invest” at all?

Your time is probably your most precious asset. When you spend time on learning, you can’t take back this time. Let’s make it count. Let’s start with two major problems I see.

Learning that doesn’t advance you toward your goals

If your goals are ambitious you will have to learn and evolve to meet them. No doubt.

But even with the best intentions you can get it wrong.

Learning wrong things for your goals

You might be asking, who would fall into this trap? It’s surprisingly easy if you don’t doenough pre-learning research or lack self awareness. What skills and knowledge will unlock your progress?

At times, it’s better to go broad than deep.Maybe you should take a UX or Project Management course instead of learning another JS framework?At other times, it’s better to go deep.

And maybe what you should do is invest in the soft skills rather than the hard skills?

Let’s take Joe as an example, Joe wants to get promoted in his software Web Development job. Joe is ambitious and willhappily spend a lot of time learning. Joe decides that they will read a book about software architecture,so they will be more ready to be a senior engineer.

What you don’t know about Joe, is that Joe is held back by a different problem. He regularly underestimates how long his tasks will take,dismisses the need for project meetings, works in long stretches without updating anybody and fails to deliver his commitments on time.

Joe doesn’t have the trust of his teammates and won’t get promoted regardless of how good his JS, Python, Rust… software architecture, distributed systems etc skills are.He needs to understand the basics of project management and become a reliable team mate.

Studying as a form of procrastination

“I just need to learn a bit more before I …”

Would you do heart surgery without training? I hope not. But for many things just a basic knowledge is enough and youwill learn much more from doing. Or you might even discover that you were learning wrong things.

Warning signs that your learning is a form of procrastination:

there isn’t a definite end or it keeps moving
you have been reading/listening to to the same type of material for a very long time (stuck in beginner mode)

The problem here might be either a lack of strategy or confidence. If it’s the strategy, put some time into thinking on a more meta level.

If it’s fear or lack of confidence, think about the actual risks you are facing:

What is at stake? E.g. bankruptcy, your life, health or maybe just a bit of an embarrassment?
Can you go back and learn more if needed without sabotaging the goal?

Maybe the risk of failure is less bad than the cost of inaction and time put into “learning”?Maybe it’s just not for you and you should move on completely.

When you think you are learning, but not really

Another reason why learning can be a big waste of time is when you are learning the right things,but it either doesn’t absorb or doesn’t stick.

Absorbing Knowledge: Poor learning habits

There are lots of ways to absorb new knowledge and skill, many of which are ineffective.

Some questions for you:

Are you absorbing the knowledge passively or actively?
How do you test your new understanding?

If you are passive and don’t test your understanding then you are likely not learning very well.You might feel like you are and you might think like doing it in a different way, would be slowerand less efficient.

Learning is changing, if nothing changes, there is no learning. Your brain has to go through the reps, you need to develop new a way of thinking.This is why learning with a project is good, or when there is a quiz or an assignment where you have to practice your knowledge.

Before engaging into studying you can look out for learning materials that are more active and if what you have is just reading material, you can stillmake it more active yourself.

Retaining Knowledge: lack of organization and process

I took plenty of notes in college. Do you know what I did with those notebooks after the courses finished?I threw them away to trash as I knew that I would probably never open them again.

As you continue learning and build on top of what you are learning, that’s probably fine, because the old material is stillthere. Similarly if you apply your skills soon after, you can maintain them and build upon them.

I did take a bunch of courses in quantum physics. They were actually pretty good, there were many assignments,math problems to solve, simulations to program, practical experiments to do.It was great, I learned a lot, I got good grades, but I don’t remember nor use almost any of it now. I work as an engineer in Software, not as a physicist.

There are many things like this where I put a lot of effort and years later it’s mostly forgotten. Could I relearn it - probably. But it’s still painful to me.

It’s painful enough to me that I started to be more strategic about my knowledge retention.

Living note taking system

I generally write things down for myself for two reasons:

to process stuff out - figure out a plan, explore a problem, etc
to make a resource I can use later

Notebooks or Google Doc documents work perfectly fine for the first goal. But in my case don’t workvery well for the second goal. It’s easy to forget about a given notebook, it’s not that easy tosearch and I just find it that I never refer back to them. In the past I tried evernote and google keep, but it would quickly turn into a mess.

And then I discovered Notion. It is visually pleasing, allows easy structuring, reorganizing and adding information (and has a lot of features!).It’s my tool for organizing information for the long term. There is a ton of materials out there about it and the tool is free for individuals.Go try it!

Process around recall (reinforcing skills)

Another trap with learning is not thinking about maintaining the skill. If you learn a skill to then use it immediately, you might be fine.But more often than not, you can find yourself forgetting the things you should have supposedly learned and getting frustrated.

Compound interest is a great concept, if you keep improving by 1% every day, you will end up ~37 times better over a year. Not 365% better, 3778% better!This is incredible! But what if you have a skill or knowledge attrition at the same time? +1% is offset by -1% (or more!) and you end up on average improving 0% per dayand 0 percent over a year and you get stuck. You might be thinking, why am I not leveling up, I keep learning!?

The good news is that retaining knowledge requires less effort than acquiring knowledge in the first place. But it’s still something that you need to beaware of, especially if the new things seem more exciting and satisfying to learn.

To be consistent, I would recommend having a process that you follow. I use Anki for Spaced Repetition.

It’s an app in which you can create cards and then practice recalling their contents. There is some art in making the cards, but you can always change them if they are not that great.If you can recall the note (or answer your question) well, then you will have a review scheduled after a longer period. If you forgot, you will review it again tomorrow. The better you remember the given card,the farther in the future it gets scheduled. I spend about 20 minutes a day every day reviewing the cards. It’s mostly foreign language vocab.

Keep on learning, but learn better

I hope you learned something reading this! Overall, I’m not an enemy of learning, I am obsessed with learningand I am always working on my skills, reading books, articles and taking courses.

But it can be a waste of time, I wasted a lot of time learning myself. I could have had more time, I could have been less stressed and busy.I could have learned more and more useful things.

The key is to not do things in an unexamined way. Be self aware, be strategic, learn about learning, improve over time and keep on learning.

Django Rest Framework Recipes

Mon, 26 Jul 2021 00:00:00 +0000

Introduction

One of my favorite tools in my app development toolkit is django rest framework (drf), that makes developing REST APIs with python and django easy and fun.It’s easy to set up, extensible and saves so much time.

The documentation of django rest framework is pretty extensive, there is a great tutorial, api docs and plenty of examples.But reality is usually a bit more complex and you need to customize your use of the framework.

In this article I’m going to share a bunch ofrecipes from my use of django rest framework taken from my latest project. They all come from just a single ViewSet (set of related api endpoints)!

At the end of the article I will show how all those bits fit together.

This article doesn’t provide any introduction to django or the django rest framework. Please read the official docs.

Piece of API used for examples

I’ve been working on a feature in the invertimo app where users can add multiple different investment accounts.The api is consumed by a react frontend and it’s customized by the need of the frontend app.

I’m using drf ViewSet to provide the following:

list endpoint ‘/accounts/’ that lists the accounts based on the model and provides a bunch of additional fields within each model
very rich detail get endpoint ‘/accounts/id/’ for retrieving detailed account data that takes additional parameters (from_date, to_date)
create and update endpoints that only allow to set or touch limited number of parameters
delete endpoint that in this case is very standard

Examples are using python type hints and are coming from a project using python 3.8 (at the time of writing). If you want to know more about setting up typechecking for django rest framework I recommend this article.

Recipes structure

The recipes are sorted from most common to more complex and are grouped together whenthey relate to a similar class of problems. I will provide a bit of a context, general motivation for using a given recipe and an example.

My app supports multiple users and they each have their own private data. I don’t want one userto see other user’s private data.

Motivation:

private data visible only to the owning user

Implementation:

filter the queryset

The best place to filter the queryset is to override get_queryset method provided by the parent class.

class AccountsViewSet(viewsets.ModelViewSet):    permission_classes = [permissions.IsAuthenticated]    serializer_class = AccountSerializer    basename = "account"    def get_queryset(self) -> QuerySet[models.Account]:        assert isinstance(self.request.user, User)        queryset = models.Account.objects.filter(user=self.request.user)        ...        return queryset

You can get the user from the self.request.user .

assert isinstance(self.request.user, User) is totally optional and done for the sake of narrowing down the type.In this case I can assert that the user is indeed a User because I enforce that the user has to the authenticated with:permission_classes = [permissions. IsAuthenticated] .

Different serializers for different methods within a viewset

Serializers in drf allow easy serialization (e.g to json) and deserialization (to native python) in your API.

By default there is one serializer class for a single ViewSet, even if it contains multiple separate views.

This is often fine, but at times you want to do it differently. See a simple example below:

Notice that there are less fields in the create form than within serialized values.

Motivation:

rich display of related fields for read only version
fields that are computed based on more complex logic

Ways to do it:

reimplement each method you want to override the serializer for (repetitive)
override get_serializer_class (recommended!)

class AccountsViewSet(viewsets.ModelViewSet):    permission_classes = [permissions.IsAuthenticated]    serializer_class = AccountSerializer    basename = "account"    def get_serializer_class(self):        if self.action in ("create", "update"):            return AccountEditSerializer        if self.action == "retrieve":            return AccountWithValuesSerializer        return AccountSerializer

If the only difference between serializers you have is that some fields are read only and shouldn’t be used in views that are updating the data,you might want to mark those fields are read only instead of changing the serializer.

Add fields on the fly that aren’t present on the model

Another common case I encountered while developing APIs was adding more data to the serialized model instances.

I present two recipes here:

using queryset annotation
using a method on a serializer

Based on queryset annotation

Motivation:

add a field based on the result SQL query, e.g. count of related entities

How to do it:

annotate the queryset
update the serializer to display new fields

Here I’m adding positions_count and transactions_count :

class AccountsViewSet(viewsets.ModelViewSet):    permission_classes = [permissions.IsAuthenticated]    serializer_class = AccountSerializer    pagination_class = LimitOffsetPagination    basename = "account"    def get_queryset(self) -> QuerySet[models.Account]:        assert isinstance(self.request.user, User)        queryset = models.Account.objects.filter(user=self.request.user).annotate(            positions_count=Count("positions", distinct=True),            transactions_count=Count("positions__transactions", distinct=True),        )        return queryset

Define additional fields within the serializer by specifying the fields and adding them to the list in the meta.New fields here are positions_count and transactions_count .

class AccountSerializer(serializers.ModelSerializer[Account]):    positions_count = serializers.IntegerField()    transactions_count = serializers.IntegerField()    class Meta:        model = Account        fields = [            "id",            "nickname",            "description",            "balance",            "last_modified",            "positions_count",            "transactions_count",        ]

The great thing about adding fields with queryset annotations is that it’s also efficient and prevents using an excessive number of SQL queries.

Useful links:

serializer fields to be included

Based on a dynamic function

Sometimes you can’t express the additional field you need this way. For example you need to call a method on the instance of the object to get the value you need.

There is an easy way with drf to do this with SerializerMethodField .

Motivation:

add a field to serialized data that requires custom logic

How to do it:

define a field as serializers.SerializerMethodField()
add it to fields list in Meta
define get_myfieldname method

Here is an example where I define new field called values :

class AccountWithValuesSerializer(serializers.ModelSerializer[Account]):    positions_count = serializers.IntegerField()    transactions_count = serializers.IntegerField()    currency = CurrencyField()    values = serializers.SerializerMethodField()    class Meta:        model = Account        fields = [            "id",            "currency",            "nickname",            "description",            "balance",            "last_modified",            "positions_count",            "transactions_count",            "values",        ]    def get_values(self, obj):        from_date = self.context["from_date"]        to_date = self.context["to_date"]        return obj.value_history_per_position(from_date, to_date)

In this example I use some additional data from self.context that brings me to my next recipe.

Pass additional data to the serializer

Motivation:

use additional data to generate and additional field
perform additional validation

How to do it:

override get_serializer_context
the data can come e.g. from self.request, e.g. self.request.user of self.request.query_params

Example from the ViewSet code:

    def get_serializer_context(self) ->  Dict[str, Any]:        context: Dict[str, Any] = super().get_serializer_context()        query = FromToDatesSerializer(data=self.request.query_params)        context["request"] = self.request        return context

and then the set value can be used inside the serializer:

request = self.context.get("request")

Use a serializer for the query parameters

Serializers transform data between formats such as json and native python, they also provide a good place toput your validation logic. Well, you can use a serializer to extract and validate data from the query parameters (also known as URL params).

Motivation:

use additional query parameters within a ViewSet and have them validated

How to do it:

define a custom Serializer for the data you expect in your query params (see that this serializer is not based on the ModelSerializer)
use it within your view:
- initialize with MySerializer(data=self.request.query_params)
- validate and extract the data

You can combine with a previous technique of passing additional data through the serializer context like follows:

class FromToDatesSerializer(serializers.Serializer[Any]):    from_date = serializers.DateField(required=False)    to_date = serializers.DateField(required=False)

Useful links:

serializers documentation

class AccountsViewSet(viewsets.ModelViewSet):    ...    def get_serializer_context(self) ->  Dict[str, Any]:        context: Dict[str, Any] = super().get_serializer_context()        query = FromToDatesSerializer(data=self.request.query_params)        context["request"] = self.request        if query.is_valid(raise_exception=True):            data = query.validated_data            self.query_data = data            context["from_date"] = self.query_data.get(                "from_date",                datetime.date.today() - datetime.timedelta(days=30),            )            context["to_date"] = self.query_data.get("to_date", datetime.date.today())        return context

Use string value in an API for a field that has more efficient DB representation (enum)

If a field can only have limited number of options, enums are a great choice.Django provides TextChoices , IntegerChoices , and Choices

enumeration types to make it very easy.

My preferred field is the IntegerChoices because it will end up using much less space in the database even if the represented value is a string.

I have a currency field defined as follows:

class Currency(models.IntegerChoices):    EUR = 1, _("EUR")    GBP = 2, _("GBP")    USD = 3, _("USD")    GBX = 4, _("GBX")def currency_enum_from_string(currency: str) -> Currency:    try:        return Currency[currency]    except KeyError:        raise ValueError("Unsupported currency '%s'" % currency)def currency_string_from_enum(currency: Currency) -> str:    return Currency(currency).labelclass Account(models.Model):    user = models.ForeignKey(User, on_delete=models.CASCADE)    currency = models.IntegerField(choices=Currency.choices, default=Currency.EUR)    nickname = models.CharField(max_length=200)    description = models.TextField(blank=True)

There are only 4 different values and they are stored very efficiently in the database.

However, I don’t want my API to expect value ‘1’ for EUR. I would much rather have “EUR” to represent “EUR” and not expose that I use integers to represent it in the database.

Motivation:

Use different internal and external representation for a value, e.g. integer vs string

How to do it:

create a new serializer Field class inheriting from a field that would be suitable for the internal representation
define to_representation and to_internal_value methods
specify that field in the serializer explicitly by using the newly defined class

class CurrencyField(serializers.IntegerField):    def to_representation(self, value):        return models.currency_string_from_enum(value)    def to_internal_value(self, value):        return models.currency_enum_from_string(value)

class AccountEditSerializer(serializers.ModelSerializer[Account]):    # Currency needs to be changed from string to enum.    currency = CurrencyField()

Useful links:

Unique together with a user that is not set in the form

It’s a fairly common case to create objects for a user that is currently logged in.But what if you don’t pass the user in the form directly? And what if you want the objects e.g. name to be unique for a given user?

You can combine:

passing additional value to the serializer with the serializer context by overriding get_serializer_context
custom field validation (override the validate_myfieldname method)

In this example the Account model has a constraint that nicknames have to be unique for a given user:

    # In the model.    class Meta:        unique_together = [["user", "nickname"]]

Within a serializer nickname field validation is overridden and if the uniqueness constraint is not satisfied,the serializer raises serializers.ValidationError.

     class Meta:        model = Account        fields = [            "id",            "currency",            "nickname",            "description",        ]    def validate_nickname(self, value):        # If user was also included in the serializer then unique_together        # constraint would be automatically evaluated, but        # since user is not included in the serializer the validation is        # done manually.        request = self.context.get("request")        if request and hasattr(request, "user"):            user = request.user            if Account.objects.filter(user=user, nickname=value).count() > 0:                raise serializers.ValidationError(                    f"User already has an account with name: '{value}'"                )        return value

Useful links:

serializer validation documentation

ViewSet using all these patterns

Well, if you are curious what is the monstrosity I’ve been working on,I’m presenting you the code of it, showcasing how all these recipes fit together.

(Tests are not included, even though they exist! The entire codebase can be found here.)

models.py :

class Currency(models.IntegerChoices):    EUR = 1, _("EUR")    GBP = 2, _("GBP")    USD = 3, _("USD")    GBX = 4, _("GBX")def currency_enum_from_string(currency: str) -> Currency:    try:        return Currency[currency]    except KeyError:        raise ValueError("Unsupported currency '%s'" % currency)def currency_string_from_enum(currency: Currency) -> str:    return Currency(currency).labelclass Account(models.Model):    user = models.ForeignKey(User, on_delete=models.CASCADE)    currency = models.IntegerField(choices=Currency.choices, default=Currency.EUR)    nickname = models.CharField(max_length=200)    description = models.TextField(blank=True)    balance = models.DecimalField(max_digits=12, decimal_places=5, default=0)    last_modified = models.DateTimeField(auto_now=True, null=True)    def __str__(self):        return (            f"<Account user: {self.user}, nickname: '{self.nickname}', "            f"currency: {self.get_currency_display()}>"        )    def value_history_per_position(self, from_date, to_date):        results = []        for position in self.positions.all():            results.append(                (                    position.pk,                    position.value_history_in_account_currency(from_date, to_date),                )            )        return results    class Meta:        unique_together = [["user", "nickname"]]

views.py

class AccountsViewSet(viewsets.ModelViewSet):    permission_classes = [permissions.IsAuthenticated]    serializer_class = AccountSerializer    pagination_class = LimitOffsetPagination    basename = "account"    def get_queryset(self) -> QuerySet[models.Account]:        assert isinstance(self.request.user, User)        queryset = models.Account.objects.filter(user=self.request.user).annotate(            positions_count=Count("positions", distinct=True),            transactions_count=Count("positions__transactions", distinct=True),        )        return queryset    def get_serializer_context(self) -> Dict[str, Any]:        context: Dict[str, Any] = super().get_serializer_context()        context["request"] = self.request        query = FromToDatesSerializer(data=self.request.query_params)        if query.is_valid(raise_exception=True):            data = query.validated_data            self.query_data = data            context["from_date"] = self.query_data.get(                "from_date",                datetime.date.today() - datetime.timedelta(days=30),            )            context["to_date"] = self.query_data.get("to_date", datetime.date.today())        return context    def get_serializer_class(        self,    ) -> Type[        Union[AccountEditSerializer, AccountWithValuesSerializer, AccountSerializer]    ]:        if self.action in ("create", "update"):            return AccountEditSerializer        if self.action == "retrieve":            return AccountWithValuesSerializer        return AccountSerializer    def retrieve(self, request, pk=None):        queryset = self.get_queryset()        queryset = queryset.prefetch_related("positions__security")        account = get_object_or_404(queryset, pk=pk)        serializer = self.get_serializer(account, context=self.get_serializer_context())        return Response(serializer.data)    def create(self, request, *args, **kwargs):        serializer = self.get_serializer(            data=request.data, context=self.get_serializer_context()        )        serializer.is_valid(raise_exception=True)        assert isinstance(self.request.user, User)        accounts.AccountRepository().create(            user=self.request.user, **serializer.validated_data        )        headers = self.get_success_headers(serializer.data)        return Response(            serializer.data, status=status.HTTP_201_CREATED, headers=headers        )

serializers.py :

class AccountSerializer(serializers.ModelSerializer[Account]):    positions_count = serializers.IntegerField()    transactions_count = serializers.IntegerField()    currency = CurrencyField()    class Meta:        model = Account        fields = [            "id",            "currency",            "nickname",            "description",            "balance",            "last_modified",            "positions_count",            "transactions_count",        ]class AccountEditSerializer(serializers.ModelSerializer[Account]):    # Currency needs to be changed from string to enum.    currency = CurrencyField()    class Meta:        model = Account        fields = [            "id",            "currency",            "nickname",            "description",        ]    def validate_nickname(self, value):        # If user was also included in the serializer then unique_together        # constraint would be automatically evaluated, but        # since user is not included in the serializer the validation is        # done manually.        request = self.context.get("request")        if request and hasattr(request, "user"):            user = request.user            if Account.objects.filter(user=user, nickname=value).count() > 0:                raise serializers.ValidationError(                    f"User already has an account with name: '{value}'"                )        return valueclass AccountWithValuesSerializer(serializers.ModelSerializer[Account]):    positions_count = serializers.IntegerField()    transactions_count = serializers.IntegerField()    currency = CurrencyField()    values = serializers.SerializerMethodField()    class Meta:        model = Account        fields = [            "id",            "currency",            "nickname",            "description",            "balance",            "last_modified",            "positions_count",            "transactions_count",            "values",        ]    def get_values(self, obj):        from_date = self.context["from_date"]        to_date = self.context["to_date"]        return obj.value_history_per_position(from_date, to_date)

Recommendations

Django rest framework is surprisingly well designed and provides a lot of great places for convenient customization.Whenever you are doing something a bit unusual, instead of jumping straight to stack overflow spend some time looking at the https://github.com/encode/django-rest-framework/tree/master either on github or within your editor.

Happy coding! If you find this article helpful, please share it and feel free to follow me on twitter.

Django - growing up from SQLite to PostgreSQL

Sun, 11 Jul 2021 00:00:00 +0000

Django by default comes with SQLite as a database.It’s a great choice for initial development and for testing,but not if you want to run multiple app servers, e.g. in a SaaS project. At some point it’s time to pick up a DB like MySQL or PostgreSQL that is more suitable for this purpose.

I’ve been working on my open source side project Invertimofor a while and at some point I decided tochange the db engine from SQLite to PostgreSQL. Well… it wasn’t an entirely smooth transition.

I’m presenting you three stories of my database migration struggles with some valuable lessons learned along the way.

Configuration glitches

I don’t set up new databases like that every day, so I looked up some tutorials and followed this one.There was only one issue, my tests would not work due to permission issues!

Thanks goodness, Django has pretty good documentation for its settings, you can find relevant sectionshere and here.

Aside from using a separate database, the test runner will otherwise use all of the same database settings you have in your settings file: ENGINE, USER, HOST, etc. The test database is created by the user specified by USER, so you’ll need to make sure that the given user account has sufficient privileges to create a new database on the system.

The key takeaways are:

when using PostgreSQL - the user needs to have Create DB permissions
test db is basically the same as the ’normal’ db, you can do very minor overrides, like change a NAME

To solve the permission problem I needed to give my PostgreSQL user permissions to create new databases:

ALTER USER myuser CREATEDB;

As a side note, I have tried a configuration like below to make my tests run with SQLite:

** BAD **

DATABASES = {    'default': {        'ENGINE': 'django.db.backends.postgresql',        'USER': 'mydatabaseuser',        'NAME': 'mydatabase',        'TEST': {            # Will not work, since Django will take the ENGINE from            # the value one level up.            # The bad thing is that it will not show you any warning,            # that the configuration is invalid.            'ENGINE': 'django.db.backends.sqlite3',            'NAME': 'mytestdatabase',        },    },}

But it didn’t work and I would generally not recommend that approach. I will explain the issues with that later in the article.

If you really needed to do that though, here is a configuration that would work:

import sysif 'test' in sys.argv:    DATABASES['default'] = {        'ENGINE': 'django.db.backends.sqlite3',        'NAME': 'mydatabase'    }

Instead of setting overrides within the ‘TEST’ dictionary, you could override the settings for your ‘default’ db directly when the program is run in the context of tests.More about this solution can be found here.

Problems with fixtures

I’m not a huge fan of database test fixtures,but they have their good use cases and django supports them pretty well. They can be a pain to maintain (hello, database changes!) and they canmake your tests slow if you load too much data.

In my Invertimo projectI’ve been using one fixture for things such as Stock Exchanges (otherwise setting them up in tests would require quite a lot of boilerplate).

Natural keys stumble

It was unclear to me why, but my old fixture that worked with SQLite wouldn’t work with PostgreSQL now. Obviously, I couldn’t let my tests to keep failing and I decided that it will be easier for me torecreate the data and create a new fixture.

Django implements two very convenient methods for dealing with fixtures: loaddata and dumpdata.

It seems like a common source of problems is the django contenttypes that are autogenerated and ‘shouldn’t be serialized’. Well, by default they are, unless you exclude them or… do a natural key trick.

Like I did, when I created my application:

python manage.py dumpdata --natural-primary > finance/fixtures/exchanges.json

Well, you shouldn’t just use a flag, because someone on the internet suggested doing that. It can lead to another types of problems.

Do better than me, read about natural keys and serialization in django before thinking about using them.

So, I created my new fixtures, they managed to load with PostgreSQL, I found another bug that was related to db differences which I will talk about later and all was fine for a while.

But then I added some more features and added more tests and I suddenly started experiencing weird integrity errors across tests. I don’t know how the django tests areinteracting with the db and I didn’t feel like reverse engineering the code (it’s a weekend side project after all and I have other things to do too…), but believe meit was very annoying.

My stack overflow searches were not yielding many results, but I couldn’t give up of course! The test with a fixture was rather important and I didn’tfeel like redoing the db setup from scratch in the test. I ended diving into my fixtures and analyzing both the errors I have been getting. All the errors wererelated to loading the serialized User and failing uniqueness constraint on primary keys.

I found something a bit odd - my serialized user didn’t have a ‘pk’ field.

{"model": "auth.user", "fields": {"password": "pbkd1234zE9lKo=", "last_login": "2021-05-23T12:28:17.174Z", "is_superuser": true, "username": "leethacker", "first_name": "", "last_name": "", "email": "leethacker@gmail.com", "is_staff": true, "is_active": true }

Odd, right? Well, I made it this way by using the --natural-primary flag.Adding a fake "pk": 122 manually to my fixture solved my problem. Oof, believe me that was a pretty frantic debugging session.

Note for my future self, ditch the contenttypes instead of using --natural-primary for this specific use case:

python manage.py dumpdata --exclude=contenttypes > my_data.json

Another option would be not using fixtures at all and save myself from all associated complexity of serializing and deserializing data.

Is PostgreSQL just so much slower than SQLite?

When I did my initial jump from PostgreSQL to SQLite (and re-dumped my fixture) my tests got significantly slower. I blamed it on PostgreSQL being ‘heavy-weight’ compared to SQLite.Don’t people like to switch their db to SQLite to speed up their test?

Well, my tests were awfully slow with PostgreSQL. I figured out why when I was having problems with my fixture integrity errors.My fixture was awfully, awfully long. Long enough for github not to want to show me the diff - this could have raised some red flags, but it didn’t.

Turns out I accidentally dumped multi-year daily stock price history of multiple stocks to a json text file.No surprise that my tests would take 10s+ when parsing such data. I cut it out and sped up my tests about 10x.

A tip for my future self:

Evaluate the autogenerated fixtures critically for unnecessary data.

DB representation differences can cause application bugs

SQL Databases share some common features, but also have their differences. Some obvious ones are query syntax data types and performance differences.Here is a good comparison between MySQL and PostgreSQL.The more RAW SQL or custom DB features your django app is using the harder a DB switch is going to be.

My app seemed to very simple when it comes to using the SQL database, no custom SQL, no fancy DB features, but I managed to hit a complex issue anyways!

Invertimo has a feature when user can import their stock transactions via a CSV file. I use pandas to load and manipulate that.

Here are some code snippets that contained a very specific bug:

transactions_data = pd.read_csv(filename)transaction_record = transactions_data_clean.iloc[x]

I get my float values and convert them to decimals, because this is how they are stored in the database:

quantity = transaction_record["Quantity"]...def to_decimal(pd_f):    return pd_f.astype(decimal.Decimal)accounts.AccountRepository().add_transaction(    account,    isin=isin,    exchange=exchange,    executed_at=executed_at,    quantity=to_decimal(quantity),    price=to_decimal(price),    transaction_costs=transaction_costs,    local_value=to_decimal(local_value),    value_in_account_currency=to_decimal(value_in_account_currency),    total_in_account_currency=to_decimal(total_in_account_currency),    order_id=order_id,)

I avoid adding duplicate transactions, by using Django’s get_or_create that should avoid adding an instance of the model if there is an instance with exactly same values already:

transaction, created = models.Transaction.objects.get_or_create(    executed_at=executed_at,    position=position,    quantity=quantity,    price=price,    transaction_costs=transaction_costs,    local_value=local_value,    value_in_account_currency=value_in_account_currency,    total_in_account_currency=total_in_account_currency,    order_id=order_id,)

Luckily I had tests for this functionality, unluckily the tests started failing after I switched the DB from SQLite to PostgreSQL.According to my tests, my duplicate prevention logic stopped working. In other words, I could end up with duplicate transactions now.

I narrowed down the issue to a problem with get_or_create. The only thing that changed was the DB, what could there be wrong?

It turns out that if I tried to import the same transaction twice a price in database vs a new one in the get_or_create would nownot be identical and another transaction would be created, this way creating a duplicate.

Why would a transaction not be deduplicated with get_or_create if the initializing values were identical? My best educated guess is that thedecimal values would not match. Could it be that pandas astype conversion was problematic?

def to_decimal(pd_f):    return decimal.Decimal(pd_f.astype(decimal.Decimal))

This code worked perfectly fine with get_or_create while using SQLite, but with PostgreSQL it was causing transaction duplicates.

The fix was to convert the Decimal argument to string first:

def to_decimal(pd_f):    return decimal.Decimal(pd_f.astype(str))

Most likely it was some type of a precision issue caused by conversion from floats, similar to this:

In [1]: price = 4.35In [2]: decimal.Decimal(decimal.Decimal(price))Out[2]: Decimal('4.3499999999999996447286321199499070644378662109375')In [3]: decimal.Decimal(str(price))Out[3]: Decimal('4.35')

Converting the argument to string first got rid of the ‘imprecision’ of the float and decimal stored in the db.Different databases don’t sore values like Decimal the same way, there are implementation differences big enough to change mycode from working to not working.

The lesson from here is to write good tests and use the same database engine for tests and production code.This bug was tricky and non-intuitive. Tools like mypy that helps withtypes in python could not save me here. I was lucky my tests caught it or otherwise I would have happily shipped the error in production.

Conclusion

Database switch can be a quite painful and a time consuming operation, but it can be done. It might be easier tojust start your app with the target database in mind to save yourself future trouble.

Otherwise make sure that you buffer some time to fix up the issues that might come up in the process.

Shooting yourself in a foot with django and recovering from it

Fri, 19 Feb 2021 20:45:59 +0000

I have recently built an API (for my side project Redeal) that turned out extremely slow. I knew that what I was building wasn’t supposed to be super fast or optimal and I was building it with python and django which aren’t really known for their speed, but I haven’t expected such atrocious performance:

Yup, you are seeing it right, it was taking about 30 seconds for all the API calls finish. Each individual call was slow and there were also multiple of them.In this post I will tell you what I did to make it more than 200x faster.

The problem I was facing wasn’t something fundamental to the tools I was using. I didn’t have to rewrite my app to a faster language like Go, or to ditch my SQL database.

I was pretty sure that the slowness in this case was self inflicted.

Django is one of my go to tools for building any sorts of web apps. I love the batteries included, so that I don’t have to reinvent the wheel over and over.The downside is that sometimes it’s way to easy to misuse the solution and end up with the app being very slow.

Where to even start?

My first instinct when I see a problem is to come up with some ideas straight off the bat.

Here is something that crossed my mind:

Should I use async generators with concurrent promises to parallelize my API calls? Should I add backend caching?

It would probably be fun to implement, but this is the wrong approach to performance debugging and at best would optimize my API calls to take about 2 seconds.Which would be still way to slow for my standards. I believed the problem was on the backend side.

The reason behind why there were so many API calls was that I was using an API with pagination and I was fetching next pages after previous ones completed.To simplify debugging I decided to make my pages huge so that I would only need one API call.

Since I was using Django Rest Framework with the custom paginator class, it was pretty easy to override and I simplified my problem to just one big API call. I could have also change the pagination options from the JavaScript side, but since in this case I controlled both, I chose to make changes on the backend side.

With the pagination change I ended up having just one slow request instead of many:

First hint: overall it was significantly faster than smaller but sequential requests.

I usually start debugging performance of django web apps by looking at what Django Debug Toolbar can tell me.Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/response and when clicked, display more details about the panel’s content.You can read how to set it up here.

So, since now I had just a single, but very slow API call, I opened it in it’s own tab.Since I was using Django Rest Framework, the JSON was rendered nicely and I could also see the Debug Toolbar on the side.

The things that immediately got my attention:

Time (5s+)
SQL (296 queries!) - each query individually was fast and

When you click at the navigation in the toolbar you can see more info in the panel.There I saw that there were extremely many similar to each other queries that were sequential and that they were taking a very long time, because of that.

Another panel that I investigated was the Profiling panel that shows how long did functions in the call stack take.

The profiling panel pointed me to two problems:

get_stats_from_queryset taking 1.7s
rest_framework/serializers.py taking 1.93s

Ouch.

Reason for the slowness

The truth is that I anticipated that the get_stats_from_queryset would be slow when I was writing it.I knew that I would have to optimize it, because I was writing some fairly naive code. It was howevera bit surprising that the serializer code was also very slow.

Let’s dig into my specific problem.I’ve been using django ORM and my models where more less as follows:

class Listing(models.Model):    url = models.URLField(unique=True)    ...    description = models.TextField()class Price(models.Model):    listing = models.ForeignKey(Listing,                                on_delete=models.CASCADE,                                related_name='prices')    created = models.DateTimeField(auto_now_add=True)    value = models.DecimalField(        max_digits=10, decimal_places=2, null=True)    class Meta:        ordering = ['-created']

I’ve been tracking listings over time and I was maintaining the history of price changes.What I also wanted in my API was price statistics for the subset of listings matching given criteria.That was computed in the function get_stats_from_queryset which result would later be added to the response theDjango Rest Framework ListAPIView would return.

Here is the listing of how the records were prepared to later get statistics:

def get_stats_from_queryset(queryset):    stats = {}    # Within the queryset, map each property to the last price.    last_prices = []    for listing in queryset:        prices = listing.prices.order_by('-created')        if prices:            last_price = prices[0].value            last_prices.append(last_price)    ...

And here is the simplified code of the serializers:

from rest_framework import serializersclass PriceSerializer(serializers.ModelSerializer):    class Meta:        model = Price        fields = ['created', 'value']class ListingSerializer(serializers.ModelSerializer):    prices = PriceSerializer(many=True)    class Meta:        model = Listing        fields = ['url', ... 'description', 'prices']

So, this isn’t anything super special, two models with a ForeignKey 1 to Many relationship.

The majority of the slowness was coming from a deluge of simple queries that were comingfrom both the serializers and the custom function that computed the stats.

Making things fast

At this point, I knew that the best way to speed up this API would be to reduce the amount of SQL queries.

I have briefly considered just writing my SQL query from scratch, but it would be surprisingly complicated to dofor selecting the value of the latest price (I have walked through it and there would be at least two joins necessary).

Luckily, this is not the new problem for the django users and django offers really good tools to optimize its ORM queries.You could use select related or prefetch related.

In this case I needed to use prefetch related, because I needed to fetch multiple related objects.I made a small tweak to the get_queryset method in my view that looked like this:

class ListingView(generics.ListAPIView):    ...    def get_queryset(self):        ...        return queryset.filter(                ... (my custom filters) ..            ).prefetch_related('prices')

I navigated back to my django rest framework API page and checked the performance:

It improved significantly! It changed from 5.7s to ~2.5s. But it was still too slow. There were still too many queries!The problem was that the get_stats_from_queryset was using listing.prices.order_by('-created') which had custom ordering and because of thatit wasn’t taking advantage of the prefetch!

This could be addressed in multiple ways.

By using specifying the ordering in the Meta

class Price(models.Model):    listing = models.ForeignKey(Listing,                                on_delete=models.CASCADE,                                related_name='prices')    created = models.DateTimeField(auto_now_add=True)    value = models.DecimalField(        max_digits=10, decimal_places=2, null=True)    class Meta:        ordering = ['-created']

(that I was already doing here)

By using

myqueryset.prefetch_related(Prefetch('prices',    queryset=Price.objects.order_by('-created')))

When either one 1. or 2. is used and if we drop the order_by from prices (replacing it with prices = listing.prices.all()), the method startstaking advantage of the prefetch_related while still remaining correct. You can both inspect the SQL queries to make sure and check if your unit tests still pass!

So, here we have it, now without the deluge of the SQL queries:

It’s still not very fast, overall the profile shows about 400ms to generate the response. Big chunk of that time comes from the rendering the fancy HTML viewfrom the Django Rest Framework that we can see on the following CPU profile:

When we request the API in the JSOn format it takes on average 230-250ms.

Making it even faster

Taking 250ms to fetch about 150 records and also compute a bunch of statistics about them it’s not a bad result. But in my mind it was still pretty slow.I further made it about 40-50% faster.

The thing was that I didn’t actually need all the prices in my API, I just needed the last price for each listing. With this mind, I denormalized my dband added a new field called last_price on each listing, which would be added to each listing. That wouldn’t be a problem because I would generally create and update the listing and prices within one method and a single transaction.I wrote a data migration in django that automatically filled in the new database field based on the prices.

https://docs.djangoproject.com/en/3.1/howto/writing-migrations/

My migration looked similar to:

from django.db import migrations, modelsdef fill_last_price(apps, schema_editor):    Listing = apps.get_model('myapp', 'Listing')    for listing in Listing.objects.all():        prices = listing.prices.order_by('-created')        if prices:            listing.last_price = prices[0].value        listing.save()class Migration(migrations.Migration):    dependencies = [        ('myapp', '0004_listing_last_price'),    ]    operations = [        migrations.RunPython(fill_last_price),    ]

With that I didn’t have to use prefetch_related and I also could simplify my serializers. It generally made my code quite a bit simpler and it was a useful tradeoff for me to make.

I went from about 30s of API calls to ~140ms.

Which is more than 200x speedup for the same exact functionality.

What went well

I found the issue in the development stage and not from the customer complaints. What helped here was that I was developing with a fairly realistic dataset. If instead of 150 records, I was using 3 in development I could have completely missed the issue. So the tip here would be to have good fixtures or development data.

The second thing that went well is that I used a disciplined approach to performance improvements starting with profiling. If I have jumped to some random idea like, concurrent promises, or even blindly applying caching I would end up with much worse results.

The last thing that I want to acknowledge is that django has a rich ecosystem and great tooling that made this process fairly easy. Yes, I shot myself in the foot at first. But django helped me recover.I would recommend any django enthusiast to use the django debug toolbar. And when it comes to the django database optimization, there is also very good documentation worth reading.

If you liked this article please share it further and follow me on twitter!

Revived

Sat, 13 Feb 2021 15:58:36 +0000

I got too busy with a new job and living in a different country. But I love writing and I am very happy to start writing in tinystruggles.com again.

What have I been up to in last couple of years? I have joined Google in Dublin, so new job, new country, new friends, new hobbies. Lot’s of changes and I got pretty absorbed by them.

Since that time (late 2015) I have updated this blog literally once and in 2017. I lost the habit of consistent blogging and I also focused my tech related activities around work. Obviously not working too much, I was pretty serious about maintaining my work life balance, but I was learning much more about distributed systems and databases instead of for example web development. I did work on bunch of technical side projects, but with Google’s copyright, etc. I felt reluctant to make any of it public.

At the end of 2019 I considered learning more about game dev and got myself a pretty powerful laptop. Nice lenovo thinkpad with a graphic card, bells and whistles. It was a nice upgrade from my at that time very old personal laptop and somehow encouraged me to start many more non-work related side projects. I also quickly decided that game dev is not my thing, but I entered 2020 enthusiastic about doing some indie-hacking. Was very productive for me when it comes to side projects, there was much less travel due to the pandemic and I finally got a really nice work from home setup.

I haven’t shared anything about them on this blog though, because I lost a good way to update it. Due to the fact that the blog was effectively abandoned (and not properly backed up!) and was using some obscure static site generation tech,I couldn’t simply pick it up and add to it. I had to actually revive it. Also, the server I was hosting it on really didn’t enjoy the update from ubuntu 14 to ubuntu 20 and broke terribly. The static site generator got migrated to python3 and was somehow also not working. It just became a terrible mess. I could maybe escape it, but it would surely take a lot of time and I wasn’t feeling like it. I just set up a new server and new blog setup to give myself a fresh start. The old posts are still accessible through here.

Mon, 01 Jan 0001 00:00:00 +0000

Marketing

Marketing to random strangers is actually really hard.

People who I talked to or interacted with are most willing to help. Build in public community on twitter. Wannabe entrepreneur folks.

Random post on twitter productivity subreddits not effective.

Getting good content for SEO not easy, hard to market productivity related content or at least I’m a bit at loss. Technical content seems easier.

How could I have credibility for productivity/life content?

Tools

canvaloomtinytestimonial

Mon, 01 Jan 0001 00:00:00 +0000