Questions and Answers :
Web site :
Collatz - something wrong today (2019-11-16)
Message board moderation
Author | Message |
---|---|
seanr22a Send message Joined: 3 Oct 19 Posts: 14 Credit: 28,576,449,285 RAC: 88,642 |
All my rigs has run empty on jobs. At two of my rigs the last 3-4 jobs error out after 1-2 sec after that no more jobs. Is this a known problem this weekend ? I did not see anything in the 'News' area on the start page about maintenance or anything. |
Bouowmx Send message Joined: 9 Jun 16 Posts: 16 Credit: 20,989,046,187 RAC: 0 |
Server status says "collatz_sieve_work_generator" is Not running, which explains that there are zero tasks ready to send. |
Tackleway Send message Joined: 29 Sep 13 Posts: 25 Credit: 3,887,559,663 RAC: 0 |
Server status says "collatz_sieve_work_generator" is Not running, which explains that there are zero tasks ready to send. I've had no new tasks since 08:39 UTC today. Definitely a problem with work generator! |
vonboedefeldt Send message Joined: 18 Oct 15 Posts: 1 Credit: 1,378,549,803 RAC: 2,113,944 |
Hallo all together, since 6:11 UTC new wus are comming. greets, vonboedefeldt |
Mastodont Send message Joined: 29 Sep 12 Posts: 3 Credit: 8,287,036,601 RAC: 614,326 |
Yes, they are coming, but computation ends with error. |
Tackleway Send message Joined: 29 Sep 13 Posts: 25 Credit: 3,887,559,663 RAC: 0 |
Yes, all new tasks error out in seconds! Needs sorting. |
![]() Send message Joined: 1 Jul 10 Posts: 14 Credit: 1,028,661,533 RAC: 502,927 |
No work yesterday. 35 Computation Errors this morning. 1 completed WU which won't upload as the server is out of space... It's going well. 17/11/2019 09:58:38 | collatz | Started upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0 17/11/2019 09:58:40 | collatz | [error] Error reported by file upload server: can't write file collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0: No space left on server 17/11/2019 09:58:40 | collatz | Temporarily failed upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0: transient upload error 17/11/2019 09:58:40 | collatz | Backing off 00:13:44 on upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0 |
KAMasud Send message Joined: 20 Oct 11 Posts: 48 Credit: 4,654,522,722 RAC: 59,019 |
Does anyone know what happened? All my tasks have started erroring out about an hour ago. |
KAMasud Send message Joined: 20 Oct 11 Posts: 48 Credit: 4,654,522,722 RAC: 59,019 |
All right, someone has heard us. Got one task running on GPU but no task for CPU. Okay, we deserve the penalty after erroring out loads of WU's. |
seanr22a Send message Joined: 3 Oct 19 Posts: 14 Credit: 28,576,449,285 RAC: 88,642 |
Got a load of tasks to all rigs now BUT of 100 tasks 97 error out after 1-2 seconds 3 go ok. Looking at server status it looks like a lot of things is down. |
![]() ![]() Send message Joined: 11 Aug 09 Posts: 963 Credit: 24,557,133,931 RAC: 68,714 |
Got a load of tasks to all rigs now BUT of 100 tasks 97 error out after 1-2 seconds 3 go ok. It's hunting season where he lives, it happens every year, in a few days he will be back home and kick the server and we will all have work again. He's tried getting others to monitor and kick it but they don't know as much as he does so it is what it is. |
![]() ![]() Send message Joined: 7 Nov 10 Posts: 5 Credit: 2,832,454,732 RAC: 0 |
I'm luckier, on 315 tasks, 222 failed (70,48%) and 93 did validate (29,52%). I would guess that remaining tasks in the scheduler are almost all broken by now. Need to get the services back online to get some fresh work. |
KAMasud Send message Joined: 20 Oct 11 Posts: 48 Credit: 4,654,522,722 RAC: 59,019 |
During the night while I was sleeping, 98% WU's errored out? |
![]() ![]() Send message Joined: 11 Aug 09 Posts: 963 Credit: 24,557,133,931 RAC: 68,714 |
During the night while I was sleeping, 98% WU's errored out? Don't know if it's fixed but the site is backup and all Servers are running again and workunits are flowing again. |
KAMasud Send message Joined: 20 Oct 11 Posts: 48 Credit: 4,654,522,722 RAC: 59,019 |
Something is still wrong, I am getting no WU's even though everything seems to be working fine server-side. Could it be possible that it is punishment for sending back donkey-cart loads of errored out WU's? |
seanr22a Send message Joined: 3 Oct 19 Posts: 14 Credit: 28,576,449,285 RAC: 88,642 |
Still something wrong. Now server status shows everything ok up and running but all WUs I receive error out. |
![]() ![]() Send message Joined: 11 Aug 09 Posts: 963 Credit: 24,557,133,931 RAC: 68,714 |
Still something wrong. Now server status shows everything ok up and running but all WUs I receive error out. That would tend to mean the workunits themselves have problems, I don't keep track enough to know if we started a new batch recently or what's going on. |
seanr22a Send message Joined: 3 Oct 19 Posts: 14 Credit: 28,576,449,285 RAC: 88,642 |
I've been away all day and checking now everything is running good now. Thanks ! |
seanr22a Send message Joined: 3 Oct 19 Posts: 14 Credit: 28,576,449,285 RAC: 88,642 |
I was a happy camper to early ... tonight tt continued with a lot of computation errors It don't error out in the same rate as yesterday but it looks like it's around 50%. I have 5 rigs with a mix NVIDIA 2080ti and 1660ti GPUs . The drivers are 441.08 on two GPUS and 441.20 on two GPUs and one on 441.12. I have not changed anything on my side since November 12 when I updated the 2080Ti GPUs to driver 441.20. The issues started this weekend when the Collatz servers had problems. I think I'm banned now because of all the errors, currently I have a 23 hour backoff for downloading new WUs. Is it only me having problems ? |
![]() ![]() Send message Joined: 11 Aug 09 Posts: 963 Credit: 24,557,133,931 RAC: 68,714 |
I was a happy camper to early ... tonight tt continued with a lot of computation errors It don't error out in the same rate as yesterday but it looks like it's around 50%. No I'm having the same problems, some units work and some just error out for me. It's also happening in both Linux and Windows machines for me. |
©2022 Jon Sonntag; All rights reserved