So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?
I’ve started using poetry and the experience has improved.
The venv stuff is pretty annoying, I agree.
As a baby Python Dev, I’m glad it’s not just me.
I’ve been full time writing python professionally since 2015. You get used to it. It starts to just make sense and feel normal. Then when you move to a different language environment you wonder why their tooling doesn’t use a virtualenv.
It… depends. There is some great tooling for Python – this was less true only a few years ago, mind you – but the landscape is very much in flux, and usage of the modern stuff is not yet widespread. And a lot of the legacy stuff has a whole host of pitfalls.
Things are broadly progressing in the right direction, and I’d say I’m cautiously optimistic, although if you have to deal with anything related to conda then for the time being: good luck, and sorry.
This is exactly how I feel about python as well… IMHO, it’s good for some advanced stuff, where bash starts to hit its limits, but I’d never touch it otherwise
It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.
I still do the python3 -m venv venv && source venv/bin/activate
How can uv help me be a better person?
And pip install -r requirements.txt
Fuck it, I just use sudo and live with the consequences.
the software equivalent of leaving the dirt on your vegetables to harden your immune system
Oh no
If you’re happy with your solution, that’s great!
uv combines a bunch of tools into one simple, incredibly fast interface, and keeps a lock file up to date with what’s installed in the project right now. Makes docker and collaboration easier. Its main benefit for me is that it minimizes context switching/cognitive load
Ultimately, I encourage you to use what makes sense to you tho :)
- let
pyproject.tomltrack the dependencies and dev-dependencies you actually care about
- dependencies are what you need to run your application
- dev-dependencies are not necessary to run your app, but to develop it (formatting, linting, utilities, etc)
- it can track exactly what’s needed ot run the application via the
uv.lockfile that contains each and every lib that’s needed. - uv will install the needed Python version for you, completely separate from what your system is running.
uv syncanduv run <application>is pretty much all you need to get going- it’s blazingly fast in everything
Thank you for explaining so clearly. Point 3 is indeed something I’ve ran into before!
- let
This! Haven’t used that one personally, but seeing how good ruff is I bet it’s darn amazing, next best thing that I used has been PDM and Poetry, because Python’s first party tooling has always been lackluster, no cohesive way to define a project and actually work it until relatively recently
I bet it’s darn amazing,
It is. In this older article (by Anna-Lena Popkes) uv is still not in the middle, but I would claim it’s the new King of Project Management, when it comes to Python.
uv init --name <some name> --package --appand you’re off to the races.Are you cloning a repo that’s
uv-enabled? Justuv syncand you’re done!Heck, you can now add dependencies to a script and just
uv run --script script.py(IIRC) and you don’t need to install anything -uvwill take care of it all, including a needed Python version.Only downside is that it’s not 1.0 yet, so the API can change at any update. That is the last hurdle for me.
I moved all our projects (and devs) from poetry to uv. Reasons were poetry’s non standard pyproject.toml syntax and speed, plus some weird quirks, e. g. if poetry asks for input and is not run with the verbose flag, devs often don’t notice and believe it is stuck (even though it’s in the default project README).
Personally, I update uv on my local machine as soon as a new release is available so I can track any breaking changes. Couple of months in, I can say there were some hiccups in the beginning, but currently, it’s smooth sailing, and the speed gain really affects productivity as well, mostly due to being able to not break away from a mental “flow” state while staring at updates, becoming suspicious something might be wrong. Don’t get me wrong, apart from the custom syntax (poetry partially predates the pyproject PEP), poetry worked great for us for years, but uv feels nicer.
Recently, “uv build” was introduced, which simplified things. I wish there was an command to update the lock file while also updating the dependency specs in the project file. I ran some command today and by accident discovered that custom dependency groups (apart from e. g. “dev”) have made it to uv, too.
“uv pip” does some things differently, in particular when resolving packages (it’s possible to switch to pip’s behavior now), but I do agree with the decisions, in particular the changes to prevent “dependency confusion” attacks.
As for the original question: Python really has a bit of a history of project management and build tools, I do feel however that the community and maintainers are finally getting somewhere.
cargo is a bit of an “unfair” comparison since its development happened much more aligned with Rust and its whole ecosystem and not as an afterthought by third party developers, but I agree: cargo is definitely a great benchmark how project and dependency management plus building should look like, along with rustup, it really makes the developer experience quite pleasant.
The need for virtual environments exists so that different projects can use different versions of dependencies and those dependencies can be installed in a project specific location vs a global, system location. Since Python is interpreted, these dependencies need to stick around for the lifetime of the program so they can be imported at runtime. poetry managed those in a separate folder in e. g. the user’s cache directory, whereas uv for example stores the virtual environment in the project folder, which I strongly prefer.
cargo will download the matching dependencies (along with doing some caching) and link the correct version to the project, so a conceptual virtual environment doesn’t need to exist for Rust. By default, rust links everything apart from the C runtime statically, so the dependencies are no longer neesed after the build - except you probably want to rebuild the project later, so there is some caching.
Finally, I’d also recommend to go and try setting up a project using astral’s uv. It handles sane pyproject.toml files, will create/initialize new projects from a template, manages virtual environments and has CLI to build e. g. wheels or source distribution (you will need to specify which build backend to use. I use hatchling), but thats just a decision you make and express as one line in the project file. Note: hatchling is the build backend, hatch is pypa’s project management, pretty much an alternative to poetry or uv.
uv will also install complete Python distributions (e. g. Python 3.12) if you need a different interpreter version for compatibility reasons
If you use workspaces in cargo, uv also does those.
uv init, uv add, uv lock --upgrade, uv sync, uv build and how uv handles tools you might want to install and run should really go a long way and probably provide an experience somewhat similar to cargo.
I think you responded to the wrong comment, I didn’t question the need for uv or other tools like that
I did that on purpose, i. e. I wanted to confirm your thoughts about uv, drifted off into a general rant, remembered OP’s original question and later realized it would have been better framed as a top level comment. In my defense, I was in an altered state of mind at the time.
Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren’t necessarily compatible.
You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
A perfect summary of the history of computer code!
venv nonsense
I mean, the fact that it isn’t more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I’d call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.
Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I’d be fine with it.
And I don’t do much Python development, so this isn’t from a “Python awesome” standpoint.
pyenv and uv let you install and switch between multiple Python versions.
As for uv, those come from the Python build standalone project, if I remember correctly, pyenv also installs from there, but don’t quote me on that.
No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault but there are a ton of options and a lot of very similarly naked things that accomplish different (but sometimes similar) tasks. As someone who considers themselves between beginner and intermediate proficiency in Python this is my biggest hurdle right now.
Python’s tooling is a mess.
Not only that. It’s a historic mess. Over the years, growing a better and better toolset left a lot of projects in a very messy state. So many answers on Stack Overflow that mention
easy_install- I still don’t know what it is, but I guess it was some kind of protouv.Every time I’m doing anything with Python I ask myself if Java’s tooling is this complicated or I’m just used to it by now. I think a big part of the weirdness is that a lot of Python tooling is tied to the Python installation whereas in Java things like Maven and Gradle are separate. In addition, I think dependencies you install get tied to that Python installation, while in Java they just are in a cache for Maven/Gradle. And in the horrible scenario where you need to use different versions of Maven/Gradle (one place I was at specifically needed Maven 3.0.3 for one project and a different for a different, don’t ask, it’s dumb and their own fault for setting it up that way) at least they still have one common cache for everything.
I guess it also helps that with Java you (often) don’t need platform specific jar files. But Python is often used as an easy and dynamic scripting interface over more performant, native code. So you don’t really run into things like “this artifact doesn’t have a 64 bit arm version for python 2” often with Java. But that’s not a fault of Python’s tooling, it’s just the reality of how it’s used.
Python’s packaging is not great. Pip and venvs help but, it’s lightyears behind anything you’re used to. My go-to is using a venv for everything.
This isn’t the answer you want, but Go(lang) is super easy to learn and has a ton of speed on python. Yes, it’s more difficult, but once you understand it, it’s got a lot going for it.
it’s also not at all relevant. go is great, but this is about python.
I’m sorry I offended you.
this is not about offense! nobody is offended. but if you ask me for help with an apple pie and i tell you to make meatballs… it’s a confusing lack of relevance.
I did lead with an appropriate request for a sidebar. I just feel the rip about context was even less appropriate. And apple cobbler would be a better comparison. Apples, just different.
it’s not though. op has issues installing programs built in python. suggesting they rebuild those programs in go is 100% an apples to meatballs comparison, and way off topic.
They should get those same programs, but for Go. I’m sure someone has made whatever they’re doing. It would work better.
You’re not wrong, but you have offended the python guys for suggesting they use something other than their toy language.
I personally look away when I find programs I want to use that are written in python. I don’t have time to play with all that BS just to run a small software on my machine. Go is my go-to (heh) but any other modern language would be fine.such a weird take.
I agree. Python is my language of choice 80% or so of the time.
But my god, it does packaging badly! Especially if it’s dependent on linking to compiled code!
Why it is like that, I couldn’t tell. The language is older than git, so that might be part of it.
However, you’re installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?
I’m no Python expert either and yeah, from an outsider’s perspective it seems needlessly confusing.
easy_installthat’s never been easy,pipthat should absolutely be put on a Performance Improvement Plan, and now thisvenvnonsense.You can criticize javascript’s ridiculous dependencies all you want (left-pad?), but one thing that they absolutely got right is how to manage them. Everything’s in
node_modulesand that’s it. Yeah, you might get eleven copies of left-pad on your system, but you know what you NEVER get? Version conflicts between projects you’re working on.Seriously. Those are EXACTLY the thoughts I had after I was forced to deal with Python after a ton of time writing projects in JS.
Yeah the tooling sucks. The only tooling I’ve liked is Poetry, I never have trouble installing or packaging the apps that use it.
Just out of curiosity, I haven’t seen anyone recommend miniconda… Why so, is there something wrong I’m not aware of?
I’m no expert, but I totally feel you, python packages, dependencies and version matching is a real nightmare. Even with
venvI had a hard time to make everything work flawlessly, especially on MacOS.However, with miniconda everything was way easier to configure and worked as expected.
Isn’t conda specifically for mathy things?
I haven’t heard of Mathy, but it seems to be a math tool?
From what I gathered, miniconda is like pipx or venv. It’s able to create python virtual environments.
But I’m very new to all of this so I’m not really a good source. However after experimenting with either of them (venv, pip or miniconda) I found miniconda the easiest to use, but that’s also probably a skill issue.
I was genuinely asking because their could be something I wasn’t aware of because yeah I’m new to all of this. (proprietary, bugs, not the right tool…
You seem related to programming, maybe you could give me some pointers here?
