Hi everyone, I need some #mastodon #admin help.
Basically, my server froze while I was running RAILS_ENV=production bundle exec rails assets:precompile for my instance update.
No response.
Hard reboot.
Instance was empty and CPU overloaded, so I just reverted to a snapshot pre-update in case I messed the DB for good.
The problem is still there, and still the same. I don't know where to start. Any help and boost is appreciated !
@tcit welp i have opened a ticket to my provider for now 😅 the server is really unstable, it may not be just me breaking the instance with the aborted update.
@shad Well, an interrupted assets:precompile is not the cause of your issues. A hard reboot could have been, but since you rolled back the database, you should be fine if the hardware is.
@tcit Thanks, at least that cheers me up :')
@shad Oh no! Do you have a remote backup of your database?
@Sylvhem i have, but it's an old one :/ i hadn't automated the process yet
@shad Shit :/.
@Sylvhem meh, at least in worst case scenario i wouldn't have to start again from scratch. Which is good, because i would not.
@shad assets:precompile requires a lot of RAM. You may want to stop every other service when running it, and consider adding some swap space.
As for the instance being empty and your CPU overloaded, that doesn't sound right. What do you mean by empty? Maybe the server is failing to start, and just starts/crashes in a tight loop?
@Thib yeah, i think i'll focus on the CPU overload for starters, then worry about the db.
I guess a way to make a more focused toot is this :
- loaded a sane snapshot of the VM running Mastodon
- instance is now not available (i have the "distressed elephant" error)
- CPU overloaded
- the top two processes are "bundle", ~98% CPU, they seem linked to mastodon services as they disappear when i stop the services.
I don't know where to go from there, since the snapshot was from two days ago and it worked perfectly fine at the time.
@shad Did you do "bundle install --without development test" to update bundle dependencies? Which ruby version are you running?
@renatolond no, i didn't update the dependencies recently. I usually go with just "bundle install". Version is :
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
(Love your hair btw)
@shad if you do any other bundle command, does it work? (like RAILS_ENV=production bundle exec rails c)
(Thanks!)
@renatolond i did get an output, so, i guess ? I'm on my phone rn tho, i'll probably check the details at home tonight.
@renatolond My instance is back and it's thanks to you !!
The output of the command mentionned some files not merged, I deleted them, the bundle process cooled itself and we're good.
Thank you !!!
@shad tu t'en es sorti·e ? T'as la ligne complète des processus bundle ? T'es dispo sur XMPP ou un truc comme ça ?
@Thib non c'est toujours la panade 😅 je tripote en ssh depuis un terminal sur mon téléphone en attendant de rentrer chez moi. J'ai riot/matrix et Discord.
@Thib Pfiou c'est bon 😅 j'ai pas bien compris mais hé c'est l'informatique...
@shad going to tag #AdminNeedHelp for extra exposure. Hopefully someone can help you!
@maloki thanks, you're an angel !
@shad
What's drive space look like?
@thegibson loads of free space, i have a 2TB storage, barely used (1%). Fun fact is the RAM is also not used (8% use right now) so i don't know what's the CPU up to 
@rtwx @thegibson @kev great, that's definitely an instance i was looking for !
@shad DM me from your admin account and I will get you an invite link for the instance.
@kev well my instance is down for now, but thanks 😅
@shad which problem ?