Tim Salzmann

@fabiangruss Wichtig ist den täglichen Zwischen-Ramen zwischen Mittag und Abendessen nicht zu vergessen!

Currently a bit quieter here - because I’m on vacation, in Japan! 🇯🇵 Starting off in Osaka and already loving it! Will continue the thread with more memories along the way 🗾

@fabiangruss @Hacks0n Thank you!

@Hacks0n I answered your email!

@fabiangruss hi，i see the app has free lifetime access，but i can't find it in the app

@fabiangruss Welcome to the dark side!

Well here we go! First tweet from this bad boy. First impressions are … kind of interesting. Anyone interested in a little review how it feels switching from a Pixel 8 Pro and around 13 years of Android usage?

@fabiangruss B is clearer than A to me. Still I find B too subtil for first time user to directly see (had to look for some seconds even tho I knew roughly what to look for). What about slightly different color coding either for the icon or entire posts even?!

Community - I need your opinion! For improved transparency, I want to differentiate between private entries and ones you shared with your loved ones. Can't decide between A or B - or is there even a better way? #buildinpublic

@roboticseabass @MarkusRyll Geometries and meshes could be (online) transformed into signed distance functions and similarly leveraged for collision avoidance. Not as flashy as an example tho 😉

Unlock the power of data-driven models in numerical optimization and optimal control with L4CasADi! :rocket: Check out our latest example showcasing how L4CasADi can optimize a collision-free trajectory through a learned Neural Radiance Field (NeRF). #OptimalControl #L4CasADi

@fabiangruss Nice!

A lot of you asked on how I did this, so here's a complete walkthrough of what I did to 1) generate a dataset using GPT-4 2) create and train a model in CreateML 3) add it to the app and use it First step Okay, let's start with the generation of the dataset. I tried different approaches but the only way I found to make it work, was to provide a sample .json file to GPT-4. I created a small .json file in the following format by hand, an array with objects that contain a text (what the user potentially enters) and the category as "label". The more examples you add the better it will understand what you want, so you better add multiple examples for each category (this image is a shortened version). Then, go to Chat-GPT and ask it for the dataset. This was how I made it work: "I have a SwiftUI App where a user can add a memory or journal entry. Most memories are about events, their friends (sometimes specifically to a certain friend), or things like relationships.... This entry should be classified into a given set of categories: event, location, notes, personal, relationship, thoughts, health I created a .json training set for an ML Model that I will integrate in my SwiftUI app. I'll be using CoreML to train a Text classification model. Add much more entries to the .json file I provided, around 30.000 - 50.000. When you are done, generate a test set with different entries." The important part here was to attach your .json file, so that it gets the structure. GPT will generate a training set and a test set for you (a validation set will be split by CreateML automatically from the training set). Second step Open CreateML, create a new project and choose text classification: Next, add the training set and the test set (to the two big +): Then, let it train for a bit. I chose BERT Algorithm because Chat GPT recommended that for natural language understanding. For me (MacBook Pro with M1Pro) training took around two hours. Third step When finished, add the file to your Xcode project to the top level (same level as ContentView and your :App). Xcode neatly automatically creates classes around the model so that it is ready to use. In code, simply call it like this: EntryClassifier is the name of the class that was generated (my model has the same name). Make sure to catch the error and potentially return a default value like I did. ----- And that's it! This obviously is only info on how to generate a data set and then use it in your app. There's so much more to figure out regarding "What is a good training set?", "What's a good size?", "Which algorithm is best for my use case?" and so on. But have fun trying out - hope it was helpful! And if you made it to the bottom, feel free to follow or share this thread :)

@fabiangruss @goingonapp Would go with the middle one

Day 21 of #SwiftUI for @goingonapp: Officially 3 weeks in! ✔ I'm continuing working on the designs for the location picker in Figma. It's shaping up but needs some final touches. ❓ Stuck on one detail in particular: Can't quite decide on the best way to place a button on the tiny info card for each location. What do you think - which is the best option? Or any other ideas? #BuildInPublic #iOSDev

@A_Reichenbach_ @MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone x.com/markusryll/sta… Check out our new framework: L4CasADi! Makes it super easy to integrate PyTorch for control applications.

@MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone @TimSalzmann I’ve been looking at integrating a neural-based MPC into my planning stack. Are there any steps that we’re harder than you expected/things I should look out for?

With Real-time Neural MPC you can efficiently integrate large, complex neural network architectures as dynamics models in an MPC-pipeline. Compared to prior implementations we can leverage neural networks with a 4000x larger parametric capacity in a 50Hz real-time framework.

@aetheru_ @MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 Great! Feel free to reach out if you have questions or improvement suggestions!

@TimSalzmann @MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 Wow thanks for making this super cool thing available for everyone, anyway congratulations on your achievement!. I’m looking forward to using this L4Casadi for my work

In our Real-time Neural MPC paper, we leverage network capacities 4000x larger in optimizations. We now release L4CasADi, which enables easy integration of PyTorch models in optimizations on CPU and GPU, supporting fast C code generation and seamless integration in Acados.

@MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 L4CasADi Github github.com/Tim-Salzmann/l… or just pip install l4casadi —no-build-isolation (Build dependencies must be installed first)

@aetheru_ @MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 Yes, correct! Any (differentiable and jit traceable) PyTorch model is supported (convolutions, transformers …). The framework is generic and does not assume a specific robotic platform or even a robotics application: Any control system that can profit from data-driven models.

@MarkusRyll @davsca1 @drmapavone @kaufmann_elia @TimSalzmann @jonarriza96 Wow super interesting, does it means that we can use this to any type of robots including car and we can use arbitrary pytorch model to represent our robot dynamics?

@PPuchaud @MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 I am working on more descriptive dummy examples involving learned dynamics and objective functions. I will share them in the next couple of days.

@PPuchaud @MarkusRyll @davsca1 @drmapavone @kaufmann_elia @jonarriza96 Prominent examples include learning the system dynamics function as in Real-Time Neural MPC, but learning the objective or the constraints is similarly supported. Generally, a data-driven L4CasADi model can replace every symbolic formulation (MX) in CasADi.

@PPuchaud @casadi_software A concrete example of this would be arxiv.org/pdf/2203.07747…

@casadi_software Could you elaborate on the perspectives of this. Which feature it would potentially unlock ? @TimSalzmann

CasADi 🤝 PyTorch github.com/Tim-Salzmann/l… Work by Tim Salzmann

@A_Reichenbach_ @MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone The two recommendations I have: Try to trade small performance degradation for large runtime gains where possible. Stick to highly optimized individual frameworks for optimization and DL operations respectively.

@aks1812 @MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone For the gradients: while there are generally no guarantees, the worst case would be in OOD scenarios. There is lots of good work going on in OOD for NN: arxiv.org/abs/2102.12567 arxiv.org/abs/2105.04471

@MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone @TimSalzmann Few questions: In your approach you have to take the gradient of the neural network with respect to the inputs. This can be very unpredictable as there is no supervision on gradient during training. 1/2

@aks1812 @MarkusRyll @davsca1 @kaufmann_elia @jonarriza96 @drmapavone Both questions are very valid! While this paper is targeting the computational and feasibility aspect with empirical results, theoretical guarantees are an interesting research avenue. If you know prior work targeting your 2. remark (with or without DL) I would be interested.

@natolambert @MarkusRyll @kaufmann_elia @drmapavone @davsca1 Hi Nathan, the paper is currently being worked over. Just had a peak at the new version - we do cite you there. Should be up on arxiv in the coming days.

@MarkusRyll @kaufmann_elia @drmapavone @TimSalzmann @davsca1 Very cool! Will have to check out how this is different from my first paper in grad school, which is very similar. arxiv.org/abs/1901.03737 Was a little surprised to not see it cited but no big deal

Our code for "Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms" is public now: github.com/TUM-AAS/ml-cas…