April 30, 2025 outage

[VW Derf]

Hi everyone,

Were back. My service provider encountered some issues with there network storage on the evening of April 30, 2025, complicated by the fact they were in the middle of a physical move of their data centre. This caused them to have to rebuld the file system which was a slow proces, but fortunately it seems all was recovered.

With that move, other servers were moved over the past couple of nights, but they are scheduled to move the main server today (May 2nd) about 17:00 PDT and that will take them several hours as well and this will result in another outage, but a lot shorter.

That said this is the first significant outage we have had since migrating away from having self hosting everything in 2020.

Anyways, welcome back.

 

[oilhammer]

Thanks, Fred.

 

[Windex]

Thanks for all you do Fred

 

[325_Guy]

Thank you!

 

[burpod]

phew!!! was getting a little worried!!

 

[Judson]

Have your costs increased as a result?

Thanks again for all you do.

 

[jimbote]

Glad to see you back.

 

[rotaecho]

I was a bit worried about this as well, as 3 days down when I checked tabs. Being an ex-IT engineer, it was making me nervous.

If you need any IT hosting help / advice or what not, I would love to help the community.

I've got 30yrs experience of running websites on UNIX/Linux using PHP/MySQL as other technologies. Currently run www.tdivanagons.com; I do all the work myself. I was locally hosted now on a VPS; dedicated servers was overkill for the traffic I get.

If you need to setup a H/A (high availability) layout even using basic DNS round-robin during any future outages/moves could be an easy setup.

Glad you're back online!

Hi everyone,

Were back. My service provider encountered some issues with there network storage on the evening of April 30, 2025, complicated by the fact they were in the middle of a physical move of their data centre. This caused them to have to rebuld the file system which was a slow proces, but fortunately it seems all was recovered.

With that move, other servers were moved over the past couple of nights, but they are scheduled to move the main server today (May 2nd) about 17:00 PDT and that will take them several hours as well and this will result in another outage, but a lot shorter.

That said this is the first significant outage we have had since migrating away from having self hosting everything in 2020.

Anyways, welcome back.

 

[JDSwan87]

Thanks Fred!

 

[csstevej]

Yeah was worried that the site was gone…..
lol kinda like going through withdrawal
I’m glad everything is sorted out and I’ve had my fix this morning when I saw the site back up.
Thanks Fred !

 

[calimustang]

Thanks Fred for getting it back on….. but why didnt you email us all to inform us? You have our email addresses during the registration process? I had an minor anxiety attack with tons of knowledge possibly lost.

Here is why,

We lost one forum with 20 years of knowledge of the police package mustangs and their service history. Gone in the drain when the owner of the forum passed away. All of us the SSP mustang owners were devastated. All the research and so forth. Gone. The wife never informed any of us as the web server dumped the whole shebang in the drain when the payments stopped.

 

[gearheadgrrrl]

Thanks! Let us know if you need any financial support.

 

[Nuje]

Thanks! Let us know if you need any financial support.

Yes - this!
Happy to help keep the lights on here.

 

[Nuje]

Friendly reminder: If you have threads in open tabs or bookmarks that you want to get back to... Save them to your hard drive first. You never know when everything could disappear. We tell people to upload images here due to uncertainty of external sites like photo*ucket, but no place is guaranteed to safeguard your data.

If you're on a Mac / Safari (edit - and Chrome, so should work for Windows users too), there's an extension called SingleFile (no association with it) that saves .html "package" that includes all of the images and embeds and hyperlinks (.pdfs do not) to a....well, single file. Like this one.

 

[P2B]

Yes - this!
Happy to help keep the lights here.

+1

Happy to contribute $ and/or expertise towards a more resilient solution.

 

[hskrdu]

Thanks Fred!

 

[calimustang]

Thats on him. I am old school. I have Twitter and all. But does he? No. I checked twitter, the last post was 2016. Same thing with facebook, the last post was 2016……..

I ran the server as a webmaster for the web developer team from 1994-1998 and always emailed everyone when it goes down and such.

hence why I am little pissed about it.

 

[oilhammer]

Fred does a lot here for nothing, and if you notice we have no ads or banners or pop up nonsense. It is my understanding that something happened that was not planned nor expected. I suggest you give him a little slack here.

 

[VW Derf]

Thanks for the kind words everyone. Not a bad idea about dusting off the Facebook account and posting something. I’ll have to remember that if we have a significant outage again.

I used to own everything with multiple servers, had VMs with replication, and status website, etc so I could post updates and such, but after moving to cloud-based solutions back in 2020, generally outages when they happened were very short, so I didn't bother with the extra expenses and complication and simplified things. Hosting costs are probably 20%, maybe even 15%, of what they used to be so that's enabling a much more sustainable path forward when it comes to controlling expenses. Plus, there is no more driving to the colocation centre to deal with hardware issues in the middle of the night which is so nice.

In the case at hand, I am still waiting for the RFO from the service provider, but it sounds like something that normally would not have been an issue was severely impact due to the fact this happened during the had taken down half of their redundancy to do a physical move. Ya, one of those rare events and thus to ensure all there was no data loss, they took their time. Yes, a 36-hour outage was long and impactful, but we survived, and the work required from me to rebuild from backups was avoided. Sending out emails would require restoring the database from backup, cobbling something together a mechanism to send out emails, etc. and initially it only looked like the event would be a few hours. Then the question is who would read them. Though Facebook as mentioned before isn’t a bad idea so at least there is a place someone can look.

As for financial support. I have always kept this site going from donations and merchandise sales and I really appreciate the support everyone has contributed over the years. It has really been the key to keeping this place we all love add free. Recently however the shipping costs for merchandise have ballooned. What used to cost me $2 to ship years ago when I first started selling stuff slowly increased to $5 a couple of years ago and in the last few months due to different shipping requirements, it is now about $10 to ship a licence plate frame. Therefore, I need to rework somethings on that side of things.

Anyways, thanks for everyone’s continued support to keep this place going, both on the financial side and especially on the sharing of information on our platform.

 

[Nuje]

Fred does a lot here for nothing, and if you notice we have no ads or banners or pop up nonsense. It is my understanding that something happened that was not planned nor expected. I suggest you give him a little slack here.

1000% this.
Go look at any other web forum and see what that experience is like - not only the junkification with ads and banners and pop-ups and all the other annoyances that have infected pretty much every corner of the web. But also just what a positive and welcoming community that has been built, fostered, and maintained here for over a quarter-century. And THEN remember what you pay for it.

Sometimes sh!t happens; you think you're gonna be spending half an hour replacing mounts on your car; four hours later, you've finally finished after having to drill out and re-thread the bolt holes for three bolt heads that broke off due to corrosion.

 

[pkhoury]

Hi everyone,

Were back. My service provider encountered some issues with there network storage on the evening of April 30, 2025, complicated by the fact they were in the middle of a physical move of their data centre. This caused them to have to rebuld the file system which was a slow proces, but fortunately it seems all was recovered.

With that move, other servers were moved over the past couple of nights, but they are scheduled to move the main server today (May 2nd) about 17:00 PDT and that will take them several hours as well and this will result in another outage, but a lot shorter.

That said this is the first significant outage we have had since migrating away from having self hosting everything in 2020.

Anyways, welcome back.

Okay, so it wasn't just me who noticed TDIclub was down. Glad it's back! Thanks also for all you do, Fred. On a segue, I'm bummed the aluminum license plate frames are NLA.

 

[pkhoury]

Sometimes sh!t happens; you think you're gonna be spending half an hour replacing mounts on your car; four hours later, you've finally finished after having to drill out and re-thread the bolt holes for three bolt heads that broke off due to corrosion.

LMAO to this - last timing belt I did on DSEL, I ended up needing a timesert for my engine mount, because it stripped, and then took the time to do the other one (which was on its way out). A 4 hour job ended up taking two days (time to get said timeserts and then the work installing them). So yeah, moral of the story - sh!t happens and it's sometimes unavoidable but always unplanned.

 

[calimustang]

1000% this.
Go look at any other web forum and see what that experience is like - not only the junkification with ads and banners and pop-ups and all the other annoyances that have infected pretty much every corner of the web. But also just what a positive and welcoming community that has been built, fostered, and maintained here for over a quarter-century. And THEN remember what you pay for it.

the forum the sspmustang.org that was created in early 1999-2000? that we lost, had zero ads or any of the junkification that you mentioned. We paid our membership as the founding members of the SSP (special service package - police package if you will) Mustang (a lot money) and its gone into the drain. There was no backup plan is: such as an standby member (2nd admin/moderator) to keep the forum alive with 20+ years of research and history of those mustangs and their war stories, lost!

Therefore, We don't want to repeat the same mistake with the 2nd ssp mustang forum which is http://www.specialservicemustang.net/ that was created mid 2000’s for other reasons which has nothing to do about the losing the original forum. The research on that one is not complete…. Sadly. We are adding on it what we have now. My heart sunk when I found out the passing of the owner.

I emphasize on this….. please have backup for the forum…… one way or another.

 

[P2B]

I emphasize on this….. please have backup for the forum…… one way or another.

Who is Fred's backup?

I volunteer.

 

[calimustang]

I nominate p2b, oilhammer, nuje, and few others. (I would support whom are the longest active members on here without being biased) I can think of others to nominate but my mind isn't in the right mode.

But right now I am working on my 2nd 2014 JSW to get her back together and fire her up and unlock the stuck rear brakes so I can move her to middle of the driveway so I can put her into service mode to replace the failed radiator (this was not expected when I parked her).

 

[Nuje]

Who is Fred's backup?
I volunteer.

I've let Fred know that I'm more than happy to contribute in whatever way to help keep the site alive and going.
No one likes to think of the "what if I get run over by a bus" situation, but for something like tdiclub with such vast and vital (for TDI owners anyway) content, it'd definitely be worthwhile to have a human backup; and backup to that backup.

 

[Windex]

Thanks for the kind words everyone. Not a bad idea about dusting off the Facebook account and posting something. I’ll have to remember that if we have a significant outage again.

I used to own everything with multiple servers, had VMs with replication, and status website, etc so I could post updates and such, but after moving to cloud-based solutions back in 2020, generally outages when they happened were very short, so I didn't bother with the extra expenses and complication and simplified things. Hosting costs are probably 20%, maybe even 15%, of what they used to be so that's enabling a much more sustainable path forward when it comes to controlling expenses. Plus, there is no more driving to the colocation centre to deal with hardware issues in the middle of the night which is so nice.

In the case at hand, I am still waiting for the RFO from the service provider, but it sounds like something that normally would not have been an issue was severely impact due to the fact this happened during the had taken down half of their redundancy to do a physical move. Ya, one of those rare events and thus to ensure all there was no data loss, they took their time. Yes, a 36-hour outage was long and impactful, but we survived, and the work required from me to rebuild from backups was avoided. Sending out emails would require restoring the database from backup, cobbling something together a mechanism to send out emails, etc. and initially it only looked like the event would be a few hours. Then the question is who would read them. Though Facebook as mentioned before isn’t a bad idea so at least there is a place someone can look.

As for financial support. I have always kept this site going from donations and merchandise sales and I really appreciate the support everyone has contributed over the years. It has really been the key to keeping this place we all love add free. Recently however the shipping costs for merchandise have ballooned. What used to cost me $2 to ship years ago when I first started selling stuff slowly increased to $5 a couple of years ago and in the last few months due to different shipping requirements, it is now about $10 to ship a licence plate frame. Therefore, I need to rework somethings on that side of things.

Anyways, thanks for everyone’s continued support to keep this place going, both on the financial side and especially on the sharing of information on our platform.

Tried to use the donate link above - not working... :-(

 

[hskrdu]

I think the club could bring in small but reasonable revenue with merchandise sales. I used to order something once a year just to send Fred some cash, and a lot of TDI owners I meet aren't active on here, but would love to have a TDIclub hat, or sticker, or shirt. I'm not suggesting Fred take this on, but we have a wealth of members who might be willing to do so/help. Items could ship from both CAN and the US, which might help in shipping costs. I know there was a post a few weeks back where someone ordered some items and didn't have any acknowledgement at all, which is a concern, and can really injure club reputation/goodwill. The other thing I do is donate a few dollars when I sell something through the club classifieds. If we passed this spirit around, we would have another source to help with a budget. Lastly, I would be in favor of US$1 charge to post in the cars/items classifieds.

 

[Windex]

Tried to use the donate link above - not working... :-(

Used the Link on the main page top right and donated.

Thanks for all you do Fred

 

[Nuje]

Ditto.

And I'll make it even a little easier.
Here's the link to donate that's on the home page (goes to a Paypal landing page, but you can easily pay with a credit / debit card or Apple Pay without needing a Paypal account or even signing in if you do have one).

And I have received email from Fred from the email address associated with that link, so I'm confident it's all properly directed.