Inspired by the very similar thread about school incidents.

  • nick@midwest.social
    link
    fedilink
    arrow-up
    2
    ·
    27 days ago

    INC-224, never forget.

    I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.

    15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.

    15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.

    I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.

    We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.

    We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D

      • mlg@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        26 days ago

        Database (thing that holds and retrieves bunch of data) broke when it reached a size of 16 Terabytes because the underlying filesystem (Thing that lets you store data on a physical disk like a hard drive or SSD) has a maximum possible size of 16 Terabytes by default (ext4)

        16 TiB is roughly 16,000 Gigabytes which is roughly 16,000,000 Megabytes

  • NeoNachtwaechter@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Our department sometimes had a few interns, most of them young and female. Usually one of them got her workplace in the boss’s room in the office and he had plenty of time to show them how things are done etc.

    One day the boss invited all staff to his house for a nice little summer barbecue. Later in the evening we recognized him being absent from the party for nearly 2 hours, and one of the interns was missing for exactly the same time.

  • CileTheSane@lemmy.ca
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Working at McDonald’s at the time. The HR manager went on bereavement leave and a replacement was brought in. The day the HR manager came back she was told she was demoted and was put as the DriveThru order taker for a couple months before finally being fired and given severance.

    A month or 2 later the old restaurant manager who was now the “Systems Manager” and in charge of all the admin tasks stopped doing unpaid overtime, so all of his duties were taken away and he was put as DriveThru order taker.
    For 3 months he came in for exactly 8 hours every day, only did order taking in DT, and left. He was still being paid his restaurant manager’s salary during this time, the new restaurant manager was in over his head and would not ask the old restaurant manager for help. Eventually the old RM left to work for a competitor working with the old HR manager.
    Apparently the owner called the competitor to scream at them for stealing his staff

  • Pennomi@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    27 days ago

    Had an executive assistant at my company who did very little if anything. Nobody knew why she was kept around and paid so much. Everyone pressured the CEO to fire her, but he strongly resisted. Eventually she was fired, but immediately threatened to sue for sexual harassment. CEO threw her a lovely settlement check despite claiming that nothing ever happened. Mmhmm.

  • CEbbinghaus@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new “current” version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.

    • MikeOxlong@lemm.ee
      link
      fedilink
      English
      arrow-up
      1
      ·
      27 days ago

      I used to work at an accounting/consulting firm who were dead set on writing business applications in VBA within Excel. The code was embedded in the notebook, and to distribute the software was sending the latest version of the Excel file. This made version control virtually impossible, and we would instead combine our work manually.

      I cannot recommend having tech-illiterate people lead software projects.

    • Vivendi@lemmy.zip
      link
      fedilink
      arrow-up
      1
      ·
      26 days ago

      Subversion has existed probably for longer than your company, the fucking managers couldn’t be arsed to read a damn book?

      • CodeMonkey@programming.dev
        link
        fedilink
        arrow-up
        1
        ·
        26 days ago

        I had a worse experience. My first internship was doing web development in ColdFusion. Why that language? Because when the company was first starting, none of the funders wanted to learn Linux/Apache administration and CF ran on Windows.

        Also, the front end development team did not have version control but shared code via a file server.

      • CEbbinghaus@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        26 days ago

        They were using SourceSafe back then. But any source control that isnt decentralised has the same problem. If the central server gets deleted so does all history

    • Artyom@lemm.ee
      link
      fedilink
      arrow-up
      1
      ·
      26 days ago

      More impressive than the fact that you saved a repo once is that the same repo still exists today with the complete git history. At the rate companies abandon products for new ones, old repos are rare.

    • Dasnap@lemmy.world
      link
      fedilink
      arrow-up
      0
      ·
      27 days ago

      Gotta respect that save. Reminds me of the Toy Story 2 assets being lost from a server failure and they were saved by one employee having a copy on their personal computer at home.

  • oleorun@real.lemmy.fan
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    27 days ago

    Worked at a place where our CIO was completely unqualified to be a leader, much less a leader in IT. She was a micromanager who took the position of “telling stakeholders” instead of “working with stakeholders” so any project she was on was really her pushing through whatever agenda she had at the time. Meanwhile her deputy CIO was stealing computer equipment from the server room but I digress…

    April fools one year and I decide to prank it up. I moved the hinges (not the door handles) of the freezer/fridge in the breakroom so that the handle and hinges were on the same side. It’s a fifteen minute job to move everything so I did it the night before the 1st.

    The next morning our hungover CIO stumbles into the breakroom and cannot get the fridge to open. After a few seconds of futile tugging on the handle, she gave up and took her lunch to her office.

    Others in the office figured it out pretty quickly and had a good chuckle.

    Later on that day CIO sends out a nastygram about pranks being unprofessional, property damage, someone was going to be in huge trouble, yadda yadda…

    But she’s not the director. The director tells her to basically fuck off, it was a funny prank, and perhaps she needed to lighten up.

    She never found out it was me.

    • frunch@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      26 days ago

      Ha!! As an appliance repair guy i learned about reversing the door hinges+handles a long time ago. It never occurred to me to use it for a prank until i was living in my apartment for a few years, and realized it really would make more sense to reverse the hinges to open the door the other way. I moved the hinges, but then it occurred to me that i can leave the handles where they were and prank all my friends when they came over. Unsurprisingly, it works! People usually would figure it out eventually but sometimes we had to intervene if they were getting too rough with it.

      I got so used to having it set up that way that once in a blue moon I’d go to open other people’s refrigerators the wrong way (not the best look for a repair tech, LOL)

  • TwoBeeSan@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Girl did dabs on break with her gf came back zonked out since she’d never smoked weed before.

    Ended up slapping manager and getting taken away by ems

    Cook got arrested at work one time when cops came to pick her up at her job. She was 4 feet tall so we joked they picked her up and carried her away. She had to use a step stool to make the soup and someone would hide the stool from her so she’d be pissed the next morning.

    Same place had a cook drinking lean and offering it to people.

    Retirement home btw

  • s3rvant@lemmy.ml
    link
    fedilink
    English
    arrow-up
    1
    ·
    27 days ago

    Previous HR was well beyond retirement age essentially working to have something to do and one day emailed all of management a spreadsheet asking us to verify our information. That sheet contained each of our full names, addresses, phone numbers, birth dates, social security number, etc.

    To my knowledge nothing of significance happened. I have my credit frozen.

    • leisesprecher@feddit.org
      link
      fedilink
      arrow-up
      1
      ·
      26 days ago

      I worked for a company that handled a ton of personal data. Pretty much every person in Germany, including addresses, bank account details, etc.

      On my first day there (fresh from university) I was given literally full read access to the entire database. And as I later found out by accident: they did not track any data exfiltration at all. I copied several gigabytes of data without anyone noticing.

      Your data is only as secure as the least motivated data broker sees fit. And that’s not very fit.

    • Tujio@lemmy.world
      link
      fedilink
      arrow-up
      0
      ·
      26 days ago

      A few years ago I asked a customer for a list of employees, so I could verify who could purchase on their account. They replied with their personnel files. Luckily it didn’t have social security numbers, but it had a LOT of personal information. Medical records, drug test results, stuff like that.

      • watersnipje@lemmy.blahaj.zone
        link
        fedilink
        arrow-up
        1
        ·
        26 days ago

        The whole workplace drug testing thing is so wild to me. An employer can actually lay claim to your bodily fluids? Absolutely mental.

        In the Netherlands, it’s very simple:

        • if there are performance problems, then you address your employee’s performance problems.
        • if there are no performance problems, then there is no problem and what your employee does in their free time is none of your business.
  • Call me Lenny/Leni@lemm.ee
    link
    fedilink
    English
    arrow-up
    1
    ·
    27 days ago

    That’s complicated to answer in my case, as nobody gets along (I’m one of the few people with a relatively stable work relation), so there’s an incident everyday, though there are also occasional ones that stand out a lot. I for some reason have a lot of bad rep without any actual cause for it and remember people storming into our operations more than once and demanding I be exiled from the place. There are two types of people in this situation whenever it has happened: those who are almost about to oblige and fulfill their wish, and me who calls authorities and ends up dealing with the situation before they can do so before everyone just forgets all that happened.

  • 5oap10116@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    My company called all lab staff “pandemic heroes” for coming in every day during the pandemic and taking on extra work to compensate for management and office staff who stayed home for years.

    Then shortly after return to office, they closed the lab and laid off all lab staff.

      • 5oap10116@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        26 days ago

        Worst part is that they did it mostly to boost the IPI right before we went public by driving down operating costs.

        We weren’t even able to buy in u til 6 months after going public and the price leveled off at 6 months

  • CMLVI@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Depends. Had a client pull a knife on me once, and another dragged me around the facility for an hour while he tried to break down a door to “kill” another client because he had stolen the change from a $5 Taco Bell gift card.

    The other incident being was a coworker harboring one of the fugitive kids at her house with her like…6 children while her husband was away in Nebraska for work. Randomly saw her in family court a year later while I was working another job, hopefully while her husband fights her for custody of the kids…

  • mozz@mbin.grits.dev
    link
    fedilink
    arrow-up
    1
    ·
    27 days ago

    Guy found a gun in the customer’s stuff

    Guy starting waving it around and playing with it, pulled the fuckin trigger, almost shot one of his coworkers

    Cops came, guy said he was moving a cabinet and it went off which obviously no one believed, somehow he wasn’t arrested, idk

    Guy was fired over the phone before he left the customer’s house

    • mozz@mbin.grits.dev
      link
      fedilink
      arrow-up
      0
      ·
      27 days ago

      Another:

      Big awful dude starts working, among other issues he was SUPER upset that the girls at the gym are allowed to have their own separate area to work out where he can’t ogle them, he felt this was grossly unfair and was angry about it

      So anyway my boss goes back to the truck to get something, at like 9 in the morning on the job site, opens up the back, the ENTIRE truck is filled with weed smoke which billows out because big awful dude is in there getting high. Boss is upset, obviously, but big awful dude is just laughing

      I think they had to finish out the day with him but the boss was definitely irritated about it

      • mozz@mbin.grits.dev
        link
        fedilink
        arrow-up
        0
        ·
        27 days ago

        Oh shit! I forgot one from another job.

        One of the busboys walked into the office, found no people and a satchel with about $30,000 in cash, picked it up and walked out, clocked out like normal, went home.

        Guy SHOWED UP TO WORK THE NEXT DAY. Just assuming I guess, they won’t have cameras or anything, if I just don’t say anything there’s no way they can know who it was and they’ll probably just move on if I play it cool.

        I guess the management was pretty aware of his level of planning skills because they had cops waiting at the restaurant at the time of his scheduled starting time and he was taken away in cuffs, presumably not to return for quite a long time.

        • MagicShel@programming.dev
          link
          fedilink
          arrow-up
          0
          ·
          27 days ago

          I guess in his defense, he knew damn well if he stopped coming to work the day after $30k went missing, they’d know it was him.

          I mean obviously the smart thing to do is not to fucking touch the money, but I’ll give the guy showing up to work the next day. It’s not like $30k is flee-to-Argentina-and-start-a-new-life money.