Stockholm AI

DL: With Annotell, get a ranked list of all the annotators based on skill and use a cutoff. So far working with images. Still have more inbound business than we can handle. #stockholmai

DL: Find the really boring, tedious tasks and just do that well. The future belongs to the collaboration between humans and machines. The data is what is important - the things you think are important really aren't. #stockholmai

DL: Figure out who's qualified for the task. The answer is not right just because a majority gave it. #stockholmai

DL: Really hard for people to know what they want beforehand. Need to ask them what is good & what is bad. Differentiates Annotell from other crowdsourcing companies. #stockholmai

DL: Getting interpretable models on agents is way easier with Bayesian statistics that deep learning. #stockholmai

DL: What is quality? If your data is really high quality, it's probably too expensive. You want data that is just OK. How many bounding boxes do you really need? #stockholmai

DL: Data is really expensive, engineers are marginal cost/cherry on top. #stockholmai

DL: estimates that 3 billion miles are needed as training data for autonomous drive #stockholmai

DL: Usually the model is wrong when you deploy it to production, so you need to have some feedback. #stockholmai

DL: But schools just teach you to select a model and optimize hyperparameters. You're just sitting there with your poorly encoded Python script and CSV file. All the important stuff comes before this. #stockholmai

DL: Deep learning frameworks are free for a reason. It's data that is valuable. DL developers are marginally useless. #stockholmai

DL: Then need to determine how labeling should work. What do you do when inter-annotator agreement is imperfect? You can never produce accuracy better than inter-annotator consistency. #stockholmai

DL: Then need to write an annotation guideline. Can other people understand your intentions from your guidelines? Personally I struggle to write a text message. #stockholmai

DL: It's really easy to pick really hard learning tasks. Always ask: how well can a human do this task? Take dermatology for instance. Dermatologists are not very consistent at this. #stockholmai

DL: Human supervised ML. Collect unlabeled data. Most collectors don't really know what's in their data. Then define your goal. Most people overload their learning task with too many subgoals. #stockholmai

DL: I teach cars how to drive, but I don't own one and don't know how to drive and kind of don't like cars. #stockholmai

DL: You all think hyperparameter tuning is this really important thing. But a lot of AI engineers are actually arguing about what exactly is asphalt in an image. #stockholmai

DL: "I'll use machine learning, because I don't know how this works!" Throw a lot of data at the problem. We need to program by example, and therefore we need to be really sure of what examples we want. #stockholmai

DL: What is a road? Defining a drivable road turns out to be problem for autonomous driving. #stockholmai

DL: High quality training data. Computers are f--- stupid. One wrong character and it breaks. We emphasize supervised learning in ML. #stockholmai