This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

  • Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
  • Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
  • Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
  • Need to read from a curl response and it’s JSON? Reach for jq.
  • Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
  • Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

  • toynbee@lemmy.world
    link
    fedilink
    arrow-up
    0
    ·
    21 days ago

    Over the last ten - fifteen years, I’ve written lots of scripts for production in bash. They’ve all served their purposes (after thorough testing) and not failed. Pretty sure one of my oldest (and biggest) is called temporary_fixes.sh and is still in use today. Another one (admittedly not in production) was partially responsible for getting me my current job, I guess because the interviewers wanted to see what kind of person would solve a coding challenge in bash.

    However, I would generally agree that - while bash is good for many things and perhaps even “good enough” - any moderately complex problem is probably better solved using a different language.

  • ShawiniganHandshake@sh.itjust.works
    link
    fedilink
    arrow-up
    0
    ·
    21 days ago

    I’ve worked in bash. I’ve written tools in bash that ended up having a significant lifetime.

    Personally, you lost me at

    reading from the database

    Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don’t exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that’s table stakes, security-wise.

    Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there’s an existing library that wraps the API you want to use in native language constructs.

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      20 days ago

      This is almost a strawman argument.

      You don’t have to shell out to a db cli. Most of them will gladly take some SQL and spit out some output. Now that output might be in some tabular format with some pretty borders around them that you have to deal with, if you are about the output within your script, but that’s your choice and so deal with it if it’s within your comfort zone to do so. Now if you don’t care about the output and just want it in some file, that’s pretty straightforward, and it’s not too different from just some cli that spits something out and you’ve redirected that output to a file.

      I’ve mentioned in another comment where if you need to accept input and use that for your queries, psql is absolutely not the tool to use. If you can’t do it properly in bash and tools, just don’t. That’s fine.

      With web API calls, same story really; you may not be all that concerned about the response. Calling a webhook? They’re designed to be a fire and forget, where we’re fine with losing failed connections. Some APIs don’t really follow strict rules with REST, and will gladly include an “ok” as a value in their response to tell you if a request was successful. If knowing that is important to the needs of the program, then, well, there you have it. Otherwise, there are still ways you can get the HTTP code and handle appropriately. If you need to do anything complex with the contents of the response, then you should probably look elsewhere.

      My entire post is not to say that “you can do everything in bash and you should”. My point is that there are many cases where bash seems like a good sufficient tool to get that simple job done, and it can do it more easily with less boilerplate than, say, Python or Ruby.

      • monk@lemmy.unboiled.info
        link
        fedilink
        arrow-up
        0
        ·
        13 days ago

        You can do everything in bash with things not written in bash, and the parts not written in bash would be alright.

  • Gamma@beehaw.org
    link
    fedilink
    English
    arrow-up
    0
    ·
    edit-2
    21 days ago

    I agree with your points, except if the script ever needs maintaining by someone else’s they will curse you and if it gets much more complicated it can quickly become spaghetti. But I do have a fair number of bash scripts running on cron jobs, sometimes its simplicity is unbeatable!

    Personally though the language I reach for when I need a script is Python with the click library, it handles arguments and is really easy to work with. If you want to keep python deps down you can also use the sh module to run system commands like they’re regular python, pretty handy

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      21 days ago

      Those two libraries actually look pretty good, and seems like you can remove a lot of the boilerplate-y code you’d need to write without them. I will keep those in mind.

      That said, I don’t necessarily agree that bash is bad from a maintainability standpoint. In a team where it’s not commonly used, yeah, nobody will like it, but that’s just the same as nobody would like it if I wrote in some language the team doesn’t already use? For really simple, well-defined tasks that you make really clear to stakeholders that complexity is just a burden for everyone, the code should be fairly simple and straightforward. If it ever needs to get complicated, then you should, for sure, ditch bash and go for a larger language.

      • MajorHavoc@programming.dev
        link
        fedilink
        arrow-up
        0
        ·
        edit-2
        21 days ago

        That said, I don’t necessarily agree that bash is bad from a maintainability standpoint.

        My team uses bash all the time, but we agree (internally as a team) that bash is bad from a maintainability perspective.

        As with any tool we use, some of us are experts, and some are not. But the non-experts need tools that behave themselves on days when experts are out of office.

        We find that bash does very well when each entire script has no need for branching logic, security controls, or error recovery.

        So we use substantial amounts of bash in things like CI/CD pipelines.

        • Gamma@beehaw.org
          link
          fedilink
          English
          arrow-up
          0
          ·
          21 days ago

          Hell, I hate editing bash scripts I’ve written. The syntax just isn’t as easy

  • jollyroberts@jolly-piefed.jomandoa.net
    link
    fedilink
    English
    arrow-up
    0
    ·
    21 days ago

    “Use the best tool for the job, that the person doing the job is best at.” That’s my approach.

    I will use bash or python dart or whatever the project uses.

  • 31337@sh.itjust.works
    link
    fedilink
    arrow-up
    0
    ·
    21 days ago

    It’s ok for very small scripts that are easy to reason through. I’ve used it extensively in CI/CD, just because we were using Jenkins for that and it was the path of least resistance. I do not like the language though.

  • locuester@lemmy.zip
    link
    fedilink
    English
    arrow-up
    0
    ·
    edit-2
    19 days ago

    What gave you the impression that this was just for development? Bash is widely used in production environments for scripting all over enterprises. The people you work with just don’t have much experience at lots of shops I would think.

    It’s just not wise to write an entire system in bash. Just simple little tasks to do quick things. Yes, in production. The devops world runs on bash scripts.

    • FizzyOrange@programming.dev
      link
      fedilink
      arrow-up
      0
      ·
      17 days ago

      Bash is widely used in production environments for scripting all over enterprises.

      But it shouldn’t be.

      The people you work with just don’t have much experience at lots of shops I would think.

      More likely they do have experience of it and have learnt that it’s a bad idea.

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      18 days ago

      I’ve never had that impression, and I know that even large enterprises have Bash scripts essentially supporting a lot of the work of a lot of their employees. But there are also many very loud voices that seems to like screaming that you shouldn’t use Bash almost at all.

      You can take a look at the other comments to see how some are entirely turned off by even the idea of using bash, and there aren’t just a few of them.

      • locuester@lemmy.zip
        link
        fedilink
        English
        arrow-up
        0
        ·
        18 days ago

        This Lemmy thread isn’t representative of the real world. I’ve been a dev for 40 years. You use what works. Bash is a fantastic scripting tool.

        • Badland9085@lemm.eeOP
          link
          fedilink
          arrow-up
          0
          ·
          18 days ago

          I understand that. I have coworkers with about 15-20 years in the industry, and they frown whenever I put a bash script out for, say, a purpose that I put in my example: self-contained, clearly defined boundaries, simple, and not mission critical despite handling production data, typically done in less than 100 lines of bash with generous spacing and comments. So I got curious, since I don’t feel like I’ve ever gotten a satisfactory answer.

          Thank you for sharing your opinion!

          • locuester@lemmy.zip
            link
            fedilink
            English
            arrow-up
            0
            ·
            18 days ago

            My #1 rule for the teams I lead is “consistency”. So it may fall back to that. The standard where you work is to use a certain way of doing things so everyone becomes skilled at the same thing.

            I have the same rule, but I always let a little bash slide here and there.

  • Kissaki@programming.dev
    link
    fedilink
    English
    arrow-up
    0
    ·
    21 days ago

    In your own description you added a bunch of considerations, requirements of following specific practices, having specific knowledge, and a ton of environmental requirements.

    For simple scripts or duck tape schedules all of that is fine. For anything else, I would be at least mindful if not skeptical of bash being a good tool for the job.

    Bash is installed on all linux systems. I would not be very concerned about some dependencies like sqlite, if that is what you’re using. But very concerned about others, like jq, which is an additional tool and requirement where you or others will eventually struggle with diffuse dependencies or managing a managed environment.

    Even if you query sqlite or whatever tool with the command line query tool, you have to be aware that getting a value like that into bash means you lose a lot of typing and structure information. That’s fine if you get only one or very few values. But I would have strong aversions when it goes beyond that.

    You seem to be familiar with Bash syntax. But others may not be. It’s not a simple syntax to get into and intuitively understand without mistakes. There’s too many alternatives of if-ing and comparing values. It ends up as magic. In your example, if you read code, you may guess that :- means fallback, but it’s not necessarily obvious. And certainly not other magic flags and operators.


    As an anecdote, I guess the most complex thing I have done with Bash was scripting a deployment and starting test-runs onto a distributed system (and I think collecting results? I don’t remember). Bash was available and copying and starting processes via ssh was simple and robust enough. Notably, the scope and env requirements were very limited.

    • palordrolap@fedia.io
      link
      fedilink
      arrow-up
      0
      ·
      21 days ago

      You seem to be familiar with Bash syntax. But others may not be.

      If by this you mean that the Bash syntax for doing certain things is horrible and that it could be expressed more clearly in something else, then yes, I agree, otherwise I’m not sure this is a problem on the same level as others.

      OP could pick any language and have the same problem. Except maybe Python, but even that strays into symbolic line noise once a project gets big enough.

      Either way, comments can be helpful when strange constructs are used. There are comments in my own Bash scripts that say what a line is doing rather than just why precisely because of this.

      But I think the main issue with Bash (and maybe other shells), is that it’s parsed and run line by line. There’s nothing like a full script syntax check before the script is run, which most other languages provide as a bare minimum.

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      21 days ago

      As one other comment mentioned, unfamiliarity with a particular language isn’t restricted to just bash. I could say the same for someone who only dabbles in C being made to read through Python. What’s this @decorator thing? Or what’s f"Some string: {variable}" supposed to do, and how’s memory being allocated here? It’s a domain, and we aren’t expected to know every single domain out there.

      And your mention of losing typing and structure information is… ehh… somewhat of a weird argument. There are many cases where you don’t care about the contents of an output and only care about the process of spitting out that output being a success or failure, and that’s bread and butter in shell scripts. Need to move some files, either locally or over a network, bash is good for most cases. If you do need something just a teeny bit more, like whether some key string or pattern exists in the output, there’s grep. Need to do basic string replacements? sed or awk. Of course, all that depends on how familiar you or your teammates are with each of those tools. If nearly half the team are not, stop using bash right there and write it in something else the team’s familiar with, no questions there.

      This is somewhat of an aside, but jq is actually pretty well-known and rather heavily relied upon at this point. Not to the point of say sqlite, but definitely more than, say, grep alternatives like ripgrep. I’ve seen it used quite often in deployment scripts, especially when interfaced with some system that replies with a json output, which seems like an increasingly common data format that’s available in shell scripting.

  • melezhik@programming.dev
    link
    fedilink
    arrow-up
    0
    ·
    20 days ago

    We are not taking about use of Bash in dev vs use Bash in production. This imho incorrect question that skirts around real problem in software development. We talk about use of Bash for simple enough tasks where code rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, data structures support, unit tests, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      20 days ago

      But not everything needs to scale, at least, if you don’t buy into the doctrine that everything has to be designed and written to live forever. If robust, scalable solutions is the nature of your work and there’s nothing else that can exist, then yeah, Bash likely have no place in that world. If you need any kind of handling more complicated than just getting an error and doing something else, then Bash is not it.

      Just because Bash isn’t designed for something you want to do, doesn’t mean it sucks. It’s just not the right tool. Just because you don’t practice law, doesn’t mean you suck; you just don’t do law. You can say that you suck at law though.

        • Badland9085@lemm.eeOP
          link
          fedilink
          arrow-up
          0
          ·
          19 days ago

          You’re speaking prophetically there and I simply do not agree with that prophecy.

          If you and your team think you need to extend that bash script to do more, stop and consider writing it in some other languages. You’ve move the goalpost, so don’t expect that you can just build on your previous strategy and that it’ll work.

          If your “problem” stems from “well your colleagues will not likely be able to read or write bash well enough”, well then just don’t write it in bash.

      • melezhik@programming.dev
        link
        fedilink
        arrow-up
        0
        ·
        edit-2
        19 days ago

        Yep. Like said - “We talk about use of Bash for simple enough tasks … where every primitive language or DSL is ok”, so Bash does not suck in general and I myself use it a lot in proper domains, but I just do not use it for tasks / domains with complexity ( in all senses, including, but not limited to team work ) growing over time …

  • esa@discuss.tchncs.de
    link
    fedilink
    arrow-up
    0
    ·
    21 days ago

    At the level you’re describing it’s fine. Preferably use shellcheck and set -euo pipefail to make it more normal.

    But once I have any of:

    • nested control structures, or
    • multiple functions, or
    • have to think about handling anything else than simple strings that other programs manipulate (including thinking about bash arrays or IFS), or
    • bash scoping,
    • producing my own formatted logs at different log levels,

    I’m on to Python or something else. It’s better to get off bash before you have to juggle complexity in it.

    • vext01@lemmy.sdf.org
      link
      fedilink
      arrow-up
      0
      ·
      20 days ago

      -e is great until there’s a command that you want to allow to fail in some scenario.

      I know OP is talking about bash specifically but pipefail isn’t portable and I’m not always on a system with bash installed.

      • esa@discuss.tchncs.de
        link
        fedilink
        arrow-up
        0
        ·
        20 days ago

        -e is great until there’s a command that you want to allow to fail in some scenario.

        Yeah, I sometimes do

        set +e
        do_stuff
        set -e
        

        It’s sort of the bash equivalent of a

        try { 
          do_stuff()
        } 
        catch { 
          /* intentionally bare catch for any exception and error */
          /* usually a noop, but you could try some stuff with if and $? */ 
        }
        

        I know OP is talking about bash specifically but pipefail isn’t portable and I’m not always on a system with bash installed.

        Yeah, I’m happy I don’t really have to deal with that. My worst-case is having to ship to some developer machines running macos which has bash from the stone ages, but I can still do stuff like rely on [[ rather than have to deal with [ . I don’t have a particular fondness for using bash as anything but a sort of config file (with export SETTING1=... etc) and some light handling of other applications, but I have even less fondness for POSIX sh. At that point I’m liable to rewrite it in Python, or if that’s not availaible in a user-friendly manner either, build a small static binary.

  • morbidcactus@lemmy.ca
    link
    fedilink
    arrow-up
    0
    ·
    20 days ago

    I’m fine with bash for ci/cd activities, for what you’re talking about I’d maybe use bash to control/schedule running of a script in something like python to query and push to an api but I do totally get using the tools you have available.

    I use bash a lot for automation but PowerShell is really nice for tasks like this and has been available in linux for a while. Seen it deployed into production for more or less this task, grabbing data from a sql server table and passing to SharePoint. It’s more powerful than a shell language probably needs to be, but it’s legitimately one of the nicer products MS has done.

    End of the day, use the right tool for the job at hand and be aware of risks. You can totally make web requests from sql server using ole automation procedures, set up a trigger to fire on update and send data to an api from a stored proc, if I recall there’s a reason they’re disabled by default (it’s been a very long time) but you can do it.

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      20 days ago

      People have really been singing praises of Powershell huh. I should give that a try some time.

      But yeah, we wield tools that each come with their own risks and caveats, and none of them are perfect for everything, but some are easier (including writing it and addressing fallovers for it) to use in certain situations than others.

      It’s just hard to tell if people’s fear/disdain/disgust/insert-negative-reaction towards bash is rational or more… tribal, and why I decided to ask. It’s hard to shake away the feeling of “this shouldn’t just be me, right?”

      • morbidcactus@lemmy.ca
        link
        fedilink
        arrow-up
        0
        ·
        20 days ago

        I have to wonder if some of it is comfort or familiarity, I had a negative reaction to python the first time I ever tried it for example, hated the indent syntax for whatever reason.

        • lurklurk@lemmy.world
          link
          fedilink
          arrow-up
          0
          ·
          edit-2
          19 days ago

          The indent syntax is one of the obviously bad decisions in the design of python so it makes sense

        • Badland9085@lemm.eeOP
          link
          fedilink
          arrow-up
          0
          ·
          19 days ago

          Creature comfort is a thing. You’re used to it. Familiarity. You know how something behaves when you interact with it. You feel… safe. Fuck that thing that I haven’t ever seen and don’t yet understand. I don’t wanna be there.

          People who don’t just soak in that are said to be, maybe, adventurous?

          It can also be a “Well, we’ve seen what can work. It ain’t perfect, but it’s pretty good. Now, is there something better we can do?”

      • some_guy@lemmy.sdf.org
        link
        fedilink
        arrow-up
        0
        ·
        20 days ago

        The nice thing about Powershell is that it was built basically now after learning all the things that previous shells left out. I’m not fluent in it, but as a Bash aficionado, I marveled at how nice it was at a previous job where we used it.

        That said, I love Bash and use it for lots of fun automation. I think you’re right to appreciate it as you do. I have no opinion on the rest.

  • flatbield@beehaw.org
    link
    fedilink
    English
    arrow-up
    0
    ·
    20 days ago

    Just make certain the robustness issues of bash do not have security implications. Variable, shell, and path evalutions can have security issues depending on the situation.

      • flatbield@beehaw.org
        link
        fedilink
        English
        arrow-up
        0
        ·
        edit-2
        20 days ago

        Bash is especially suseptable. Bash was intended to be used only in a secure environment including all the inputs and data that is processed and including all the proccess on the system containing the bash process in question for that matter. Bash and the shell have a large attack surface. This is not true for most other languages. It is also why SUID programs for example should never call the shell. Too many escape options.

        • Badland9085@lemm.eeOP
          link
          fedilink
          arrow-up
          0
          ·
          19 days ago

          Good point. It’s definitely something to keep in mind about. It’s pretty standard procedure to secure your environments and servers, wherever arbitrary code can be ran, lest they become grounds for malicious actors to use your resources for their own gains.

          What could be a non-secure environment where you can run Bash be like? A server with an SSH port exposed to the Internet with just password authentication is one I can think of. Are there any others?

          • flatbield@beehaw.org
            link
            fedilink
            English
            arrow-up
            0
            ·
            edit-2
            19 days ago

            I was more thinking of the CGI script vunerability that showed up a few years ago. In that case data came from the web into the shell environment uncontrolled. So uncontrolled data processing where the input data crosses security boundaries is an issue kind of like a lot of the SQL injection attacks.

            Another issue with the shell is that all proccesses on the system typically see all command line arguments. This includes any commands the shell script runs. So never specify things like keys or PII etc as command line arguments.

            Then there is the general robustness issue. Shell scripts easy to write to run in a known environment and known inputs. Difficult to make general. So for fixed environment and known and controlled inputs that do not cross security boundaries probaby fine. Not that, probablay a big issue.

            By the way, I love bash and shell scripts.

          • flatbield@beehaw.org
            link
            fedilink
            English
            arrow-up
            0
            ·
            edit-2
            19 days ago

            By the way, I would not consider logging in via ssh and running a bash script to be insecure in general.

            However taking uncontrolled data from outside of that session and injecting it could well be insecure as the data is probably crossing an important security boundary.

  • synae[he/him]@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    0
    ·
    20 days ago

    As I’ve matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won’t come knocking down your door with forces and pitchforks…

    • Badland9085@lemm.eeOP
      link
      fedilink
      arrow-up
      0
      ·
      20 days ago

      That’s my take on the use of bash too. If it’s something that people think it’s worth bring their pitchforks out for, then it’s something you should probably not write in bash.