TSD Operational Log - Page 14

Dear TSD-users,

The File Lock service is down and we are working to resolve this issue.

We apologize for any inconvenience this may cause you.

Best regards,

TSD-Team

Dear TSD-Linux users,

There will be a downtime of the TSD-Linux infrastructure, during which we will reboot our ThinLinc servers to upgrade their kernel.

We apologize for any inconvenience this may cause you.

Best regards,

TSD-Team

We are facing an issue with the mount and working on a fix

The upgrade and root is taking longer then expected. We are now rebooting the last machines, expecting to be finished by this evening (around 18:00).

Please check the operational log later today.

-------------------

Due to a security vulnerability discovered in the linux RED HAT kernel, the linux machines will be rebooted on Thursday 29/06 at 14:00 CET (one hour).  All the processes running on the machines will die, so we strongly recommend to stop all the programs/processes running locally on the machine before the maintenance.
We apologise for the inconvenience this might cause to you.

We have finished the maintenance of the Colossus cluster. The outcome of this outage is that the HugeMem node will be much cheaper! Please read the post in  the "News".

We apologize for the inconvenience.

Francesca@TSD

UPDATE: Databases back up, and upgrade postponed. Our apologies for the inconvenience.

The new downtime windows are as follows.

Tuesday, 20th of June, 08:00 - 14:00
p11, p22, p23, p33, p38 and p40.

Wednesday, 21st of June, 08:00 -14:00
p47, p58, p76, p96, p158, p175 and p189

Thursday, 22nd of June, 08:00 - 09:30
p32, p225 and p244

We will update this message to keep you posted on the status of the upgrades.

Best regards,
TSD-team

We are doing some maintenance work on Colossus on Monday 12/06 at 8:00 am CET for the entire day. Jobs that will be schedule during the downtime or short before will be put on queue and will start after the maintenance.

We apologize for the inconvenience this might cause to you.

Regards,

Francesca@TSD

Dear TSD users,

Regretfully, the planned downtime is prolonged until further notice. Our experts are working on the case, and the TSD-services will be available fairly soon.

We apologize for any inconvenience this may cause.
 

Best Regards,
TSD-Team

Dear TSD users,

Dear TSD users,

We are getting into the hot part of the configuration/deployment of the new gateway machines. We need to put them in production and therefore we need a downtime of the TSD. The downtime will be on the 08/06 at 12:00 CET and will last for 3 hours. During the downtime the login to TSD will not be possible. The jobs on Colossus will keep on running. But the linux VMs most likely will need a reboot at the end of the maintenance.

We apologise for this outage with such a short notice but we are doing it for the good of the entire infrastructure.

Due to a failure of the main gateway, the TSD is not reachable at the moment.

We are working extra time to solve the problem as soon as possible.

We apologize for the inconvenience.submissions

03/06 21:46 - Login issue fixed. But still issues with some linux VMs, module load and job submission to Colossus may face problems.

04/06 -12:01 - TSD back to normal and Linux VMs can use "module load"

Affects - Linux Vms, job submission,listing or any other operation on /cluster, 

We are experiencing a mounting issue of the /cluster. This will lead to problems when submitting jobs to Colossus. The cost command may also not work.

We are working on this (07:40, 02/06) and sorry for the inconvenience.

NB NB : We did find a redhat rtcbind-bug and this is what caused the crash due to our usage of it. We have downgraded and this problem should now be totally fixed.

10:05, 02/06 : Most VMs are fixed. Still testing

Some of our Linux VMs are currently having issues mounting a network drive. As a result of this, logging in to the affected machines will not work.
We're hard at work to solve this as quickly as possible. Our apologies for the inconvenience.

UPDATE: Looks like it is possible to log on to some of the VMs, but these are still having trouble reaching the /cluster area.

-- 
Best regards,
TSD-team

Due to a jump host failure earlier this morning, some of the Linux-VMs inside of TSD require reboot and we are progressively fixing the machines now.

Our apologies for the inconvenience.

Users may currently face problems importing/exporting files to/from TSD. We are doing our best to get this issue resolved ASAP

TSD@USIT

The log onto the linux machines in the TSD infrastructure is not possible at the moment. We are working to solve the situation as soon as possible.

We apologize for the inconvenience.

Due to a jump host failure, TSD was not accessible 09:15 - 14:31 today. The failure has also caused NFS hangs on some Linux VMs, due to which we had to reboot those VMs. All thinlinc VMs now should be accessible. Some other Linux VMs my still have problems. We are working in fixing those ASAP

TSD@USIT

Due to NFS hangs, some Linux hosts are now inaccessible. The issue may result in TSD being inaccessible via thinlinc. This should not affect log in through VMWare Horizon.

We doing our best for this issue to be resolved ASAP

TSD@USIT

---------------------------------------------------

The issue was resolved after the affected hosts have been rebooted

There will be a maintenance stop on the 02/05 at 13:00 CET for one hour. During the downtime, the linux and windows VMs will not be accessible. The Colossus cluster will be under maintenance.
 

We are having an issue with the main gateway and the login to TSD in this moment is not possible. We are working to solve the problem as soon as possible.

We apologize for the inconvenience.

----

Update 14/04, 12:15 -- TSD partially operational. Still not possible to submit jobs to Colossus

----

Update 18/04, 09:40 -- Still there might be issues when submitting jobs to  Colossus

The problem was solved on Friday around 22:00. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

----

TSD is unreachable at this moment. We are trying to solve the problem as soon as possible. We apologize for the inconvenience.

The TSD gateway was not reachable between 15:43 and 16:15 today, the issue has been resolved and users should be able to log in again.

Some linux machines might still not be available, if you are unable to log into ThinLinc please contact us.

The problem was solved the same day around 18:10. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

----------------

TSD's main gateway is not reachable at the moment and the TSD infrastructure is not accessible. We are trying to solve the situation as soon as possible. More information will appear on the operational log on Monday 27/02 in the morning.

We apologize for the inconvenience.

The problem was solved on Friday around 17:15. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

------------------

TSD is unreachable at this moment. We are trying to solve the problem as soon as possible. We apologize for the inconvenience.