Michael Kocher

Kicking the tires on creating CLI tools with #ZIO and scala-cli. I really enjoy the core compositional ethos of ZIO. scala-cli is a very welcomed and promising addition to the #scala community. There's a lot of potential here. mpkocher.github.io/2022/06/02/CLI… scala-cli.virtuslab.org

@karpathy Here's a list of other Python packages that are leveraging type hint/annotations to generate command line apps. Several of these projects support loading from config files formats (e.g., json,toml,yaml). #other-related-tools" target="_blank" rel="nofollow noopener">github.com/mpkocher/pydan…

@karpathy I put a thin layer on Pydantic to generate a command line interface while also supporting JSON. It has a mix'n match model of loading config using JSON "presets" as complete/partial models and also enabling overriding these values as CLI options/flags. #loading-configuration-using-json" target="_blank" rel="nofollow noopener">github.com/mpkocher/pydan…

The software engineering aspect of deep learning repos I've been watching closely is how they store, catalogue, override, manage and plumb hyperparameter configs. Have come to dislike argparse, YAMLs (too inflexible), and fully enumerated kwargs on classes/defs. Any favorites?

@driscollis @realpython Alternatively, a range based solution could be used. I believe each `in` range call should be O(1) for non-large numbers. Also, it's probably better to use Tuple[int, ...] for breakpoint and avoid using a default mutable argument (e.g. list).

Have you tried the `bisect` module in #Python 🐍 ? #PythonStdLib 🔥 According to the docs, "this module provides support for maintaining a list in sorted order without having to sort the list after each insertion." Here's an example of using bisect to calculate letter grades:

@treyhunner It's worth noting that there are some issues when subclassing Path. github.com/python/cpython… Looks like there's an active PR that will resolve this friction point in a future release of Python. github.com/python/cpython…

I love Python's pathlib module. 💖🐍🗃 Whenever I need to ask a question about a file path in #Python, I default to using pathlib. Let's talk about some of my most common uses for pathlib and compare them to their non-pathlib alternatives. 🧵

@raymondh The "returns" package explores this space and attempts to be typesafe. While these are interesting experiments, I believe there's too much friction trying to resolve the impedance mismatch of a Result return type with the standard raise model in Python. #result-container" target="_blank" rel="nofollow noopener">github.com/dry-python/ret…

My new favorite #Python helper function makes a call and puts the results in an (exception, result) pair: >>> exc_and_value('abcde'.index, 'd') (None, 3) >>> exc_and_value('abcde'.index, 'f') (<class 'ValueError'>, None) 1/

@treyhunner You can also drag pathlib into mix.

The upside of an old programming language? Stability. The downside? Lots of ways to do the same thing. 😬 #Python os.system("ps") out = os.popen("ps").read() subprocess.call(["du", "-s"]) subprocess.run(["du", "-s"]) subprocess.run("du -s", shell=True)

@nedbat @realpython @SnickerDev It's useful to consider using itertools.takewhile for these use cases. Also, the design choice of adding a stopping sentinel and callable mechanic to iter results in a noticeable increase in the complexity of the type signature. github.com/python/typeshe…

The two-argument form of #python's iter() repeatedly gets values from a function until it finds the specified sentinel value. #iter" target="_blank" rel="nofollow noopener">docs.python.org/3/library/func… /ht @SnickerDev

@driscollis If your unittest cases are heavily parameterized, it can be useful to write a util func and have a more declarative style. This can sometimes reduce the boilerplate/duplication and improve readability.

Let's talk about test-driven development (TDD) with #Python today. We will do that by going over a code kata example! 🧵🐍🔥👇

@bernhardsson There's a ton of entropy in this space. Leaky abstractions, PBS/Slurm/SGE lack of standardization, never ending converting file format/standard X to Y, scrappy CLI tools, random grep/awk tasks, file chunking based on workflow specific data access patterns. It's a tricky space.

I get the impression that bioinformatics has its whole own ecosystem of workflow scheduling and scaling (stuff like Nextflow and SLURM) – why is that?

@raymondh Using %paste might be a useful (temporary) workaround for some cases. Paste appears to go outside of the reformatting mechanism. Using %load will still be reformatted. It's a bit unclear why %paste and %load behave differently with respect to formatting.

Argh! Who thought Black should be automatically applied to lines in the IPython CLI? The makes it less useful for education purposes, less useful for interactive math, and annoying when it rewrites your input across multiple lines.

@raymondh I believe EllipsisType has a similar friction point. In 3.10, types.EllipsisType and types.NoneType were added. #types.EllipsisType" target="_blank" rel="nofollow noopener">docs.python.org/3/library/type…

A little birdie told me that #Python typing Optional and Union are going away. It's all or-operators from here: def f(x: A | B, y = int | None = None) -> dict[tuple[str, str]]: ... I won't miss Union, but Optional was pleasantly readable. f(x: A | B, y: Optional[int] = None):

@bbelderbos The state of the cache can be obtained using .cache_info() and the cache can be cleared using .cache_clear(). The source of lru_cache is also worth reading. It has tasteful use of closures and utilizes bound functions to help optimization the algo. #L524" target="_blank" rel="nofollow noopener">github.com/python/cpython…

For caching (memoization) in #Python, check out functools.lru_cache:

@treyhunner Also note that reduce has a simple type signature (at least relative to max/min).

@treyhunner Reduce can be useful when you start to hit limitations of using key= mechanism in max/min when comparing two entities. Reduce can also be used to construct a compose function to capture the steps of a data processing pipeline. github.com/mpkocher/Funct…

I rarely use functools.reduce in #Python. Most of my reductions can be done with another tool: - adding (sum) - multiplying (math.​prod) - counting values (collections.Counter, list.count, etc) - checking conditions (any & all) What do you use (or not) functools.reduce for?

@driscollis @realpython @pybites It can be useful to leverage a custom default factory when defining dataclasses. The post_init hook in your example might be better if you use the channels that were passed in to the constructor instead of hard coding these to an empty list.

I believe I asked about how to catch attribute changes here on Twitter a few months ago. I eventually figured it out with the help of some kind people in the @realpython and @pybites communities. This article shows an example of what I learned blog.pythonlibrary.org/2021/11/10/how…

@1st1 For simple cases, it can be useful to replace if/elif blocks with a dict to perform the dispatch handling.

Recently I was staring at a series of long if..elif..elif chains in #python code and came up with a little abstraction to make the code look nicer. Please welcome value dispatch. It's just like the functools.singledispatch decorator but works on values, not types. 🐍

@anthonypjshaw Are there any optimizations in Pyjion at the JIT compiler level that leverage type annotations/hints?

@bbelderbos It can be useful to be make sure Pydantic's core design principles are consistent with your app requirements prior to adoption. Pydantic has an opinionated approach to casting/conversion. This is often useful, but sometimes can yield surprising results. #data-conversion" target="_blank" rel="nofollow noopener">pydantic-docs.helpmanual.io/usage/models/#…

Pydantic validators are wicked! From the docs: pydantic-docs.helpmanual.io/usage/validato…

@noahsbwilliams For this use case, random.choice can be helpful. #random.choice" target="_blank" rel="nofollow noopener">docs.python.org/3/library/rand…