Collatz - something wrong today (2019-11-16)

Questions and Answers : Web site : Collatz - something wrong today (2019-11-16)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
seanr22a

Send message
Joined: 3 Oct 19
Posts: 14
Credit: 13,898,704,321
RAC: 7,456,646
Message 2037 - Posted: 16 Nov 2019, 19:21:22 UTC

All my rigs has run empty on jobs. At two of my rigs the last 3-4 jobs error out after 1-2 sec after that no more jobs. Is this a known problem this weekend ? I did not see anything in the 'News' area on the start page about maintenance or anything.
ID: 2037 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bouowmx

Send message
Joined: 9 Jun 16
Posts: 7
Credit: 9,022,878,469
RAC: 18,190,987
Message 2038 - Posted: 16 Nov 2019, 20:12:22 UTC

Server status says "collatz_sieve_work_generator" is Not running, which explains that there are zero tasks ready to send.
ID: 2038 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tackleway

Send message
Joined: 29 Sep 13
Posts: 24
Credit: 3,231,906,620
RAC: 1,757,341
Message 2039 - Posted: 16 Nov 2019, 20:53:20 UTC - in response to Message 2038.  

Server status says "collatz_sieve_work_generator" is Not running, which explains that there are zero tasks ready to send.


I've had no new tasks since 08:39 UTC today. Definitely a problem with work generator!
ID: 2039 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
vonboedefeldt

Send message
Joined: 18 Oct 15
Posts: 1
Credit: 732,348,002
RAC: 373,469
Message 2040 - Posted: 17 Nov 2019, 6:32:15 UTC

Hallo all together,
since 6:11 UTC new wus are comming.
greets,
vonboedefeldt
ID: 2040 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Mastodont

Send message
Joined: 29 Sep 12
Posts: 3
Credit: 5,211,812,692
RAC: 6,715,868
Message 2041 - Posted: 17 Nov 2019, 7:50:20 UTC - in response to Message 2040.  

Yes, they are coming, but computation ends with error.
ID: 2041 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tackleway

Send message
Joined: 29 Sep 13
Posts: 24
Credit: 3,231,906,620
RAC: 1,757,341
Message 2042 - Posted: 17 Nov 2019, 8:48:18 UTC

Yes, all new tasks error out in seconds! Needs sorting.
ID: 2042 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile entigy

Send message
Joined: 1 Jul 10
Posts: 13
Credit: 655,068,368
RAC: 1,182,150
Message 2043 - Posted: 17 Nov 2019, 10:02:19 UTC

No work yesterday.
35 Computation Errors this morning.
1 completed WU which won't upload as the server is out of space...
It's going well.

17/11/2019 09:58:38 | collatz | Started upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0
17/11/2019 09:58:40 | collatz | [error] Error reported by file upload server: can't write file collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0: No space left on server
17/11/2019 09:58:40 | collatz | Temporarily failed upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0: transient upload error
17/11/2019 09:58:40 | collatz | Backing off 00:13:44 on upload of collatz_sieve_2ff79ab9-6799-4341-9903-61bca26a8f23_0_r288266730_0
ID: 2043 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
KAMasud

Send message
Joined: 20 Oct 11
Posts: 10
Credit: 484,909,518
RAC: 2,181,714
Message 2044 - Posted: 17 Nov 2019, 10:17:54 UTC

Does anyone know what happened? All my tasks have started erroring out about an hour ago.
ID: 2044 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
KAMasud

Send message
Joined: 20 Oct 11
Posts: 10
Credit: 484,909,518
RAC: 2,181,714
Message 2045 - Posted: 17 Nov 2019, 10:20:59 UTC

All right, someone has heard us. Got one task running on GPU but no task for CPU. Okay, we deserve the penalty after erroring out loads of WU's.
ID: 2045 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
seanr22a

Send message
Joined: 3 Oct 19
Posts: 14
Credit: 13,898,704,321
RAC: 7,456,646
Message 2048 - Posted: 17 Nov 2019, 11:40:05 UTC

Got a load of tasks to all rigs now BUT of 100 tasks 97 error out after 1-2 seconds 3 go ok.

Looking at server status it looks like a lot of things is down.
ID: 2048 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 11 Aug 09
Posts: 732
Credit: 22,079,867,274
RAC: 663,113
Message 2050 - Posted: 17 Nov 2019, 12:43:34 UTC - in response to Message 2048.  

Got a load of tasks to all rigs now BUT of 100 tasks 97 error out after 1-2 seconds 3 go ok.

Looking at server status it looks like a lot of things is down.


It's hunting season where he lives, it happens every year, in a few days he will be back home and kick the server and we will all have work again.
He's tried getting others to monitor and kick it but they don't know as much as he does so it is what it is.
ID: 2050 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [AF>Dell>UniversdeDell.com] Athar

Send message
Joined: 7 Nov 10
Posts: 2
Credit: 902,259,298
RAC: 817,904
Message 2053 - Posted: 17 Nov 2019, 19:42:22 UTC

I'm luckier, on 315 tasks, 222 failed (70,48%) and 93 did validate (29,52%). I would guess that remaining tasks in the scheduler are almost all broken by now.
Need to get the services back online to get some fresh work.
ID: 2053 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
KAMasud

Send message
Joined: 20 Oct 11
Posts: 10
Credit: 484,909,518
RAC: 2,181,714
Message 2058 - Posted: 18 Nov 2019, 3:49:12 UTC

During the night while I was sleeping, 98% WU's errored out?
ID: 2058 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 11 Aug 09
Posts: 732
Credit: 22,079,867,274
RAC: 663,113
Message 2061 - Posted: 18 Nov 2019, 4:30:32 UTC - in response to Message 2058.  

During the night while I was sleeping, 98% WU's errored out?


Don't know if it's fixed but the site is backup and all Servers are running again and workunits are flowing again.
ID: 2061 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
KAMasud

Send message
Joined: 20 Oct 11
Posts: 10
Credit: 484,909,518
RAC: 2,181,714
Message 2064 - Posted: 18 Nov 2019, 6:52:32 UTC - in response to Message 2061.  

Something is still wrong, I am getting no WU's even though everything seems to be working fine server-side. Could it be possible that it is punishment for sending back donkey-cart loads of errored out WU's?
ID: 2064 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
seanr22a

Send message
Joined: 3 Oct 19
Posts: 14
Credit: 13,898,704,321
RAC: 7,456,646
Message 2065 - Posted: 18 Nov 2019, 8:05:32 UTC

Still something wrong. Now server status shows everything ok up and running but all WUs I receive error out.
ID: 2065 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 11 Aug 09
Posts: 732
Credit: 22,079,867,274
RAC: 663,113
Message 2070 - Posted: 18 Nov 2019, 11:33:07 UTC - in response to Message 2065.  

Still something wrong. Now server status shows everything ok up and running but all WUs I receive error out.


That would tend to mean the workunits themselves have problems, I don't keep track enough to know if we started a new batch recently or what's going on.
ID: 2070 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
seanr22a

Send message
Joined: 3 Oct 19
Posts: 14
Credit: 13,898,704,321
RAC: 7,456,646
Message 2075 - Posted: 18 Nov 2019, 15:50:55 UTC

I've been away all day and checking now everything is running good now. Thanks !
ID: 2075 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
seanr22a

Send message
Joined: 3 Oct 19
Posts: 14
Credit: 13,898,704,321
RAC: 7,456,646
Message 2082 - Posted: 19 Nov 2019, 8:09:03 UTC

I was a happy camper to early ... tonight tt continued with a lot of computation errors It don't error out in the same rate as yesterday but it looks like it's around 50%.

I have 5 rigs with a mix NVIDIA 2080ti and 1660ti GPUs . The drivers are 441.08 on two GPUS and 441.20 on two GPUs and one on 441.12.

I have not changed anything on my side since November 12 when I updated the 2080Ti GPUs to driver 441.20.

The issues started this weekend when the Collatz servers had problems. I think I'm banned now because of all the errors, currently I have a 23 hour backoff for downloading new WUs.

Is it only me having problems ?
ID: 2082 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 11 Aug 09
Posts: 732
Credit: 22,079,867,274
RAC: 663,113
Message 2084 - Posted: 19 Nov 2019, 12:13:21 UTC - in response to Message 2082.  

I was a happy camper to early ... tonight tt continued with a lot of computation errors It don't error out in the same rate as yesterday but it looks like it's around 50%.

I have 5 rigs with a mix NVIDIA 2080ti and 1660ti GPUs . The drivers are 441.08 on two GPUS and 441.20 on two GPUs and one on 441.12.

I have not changed anything on my side since November 12 when I updated the 2080Ti GPUs to driver 441.20.

The issues started this weekend when the Collatz servers had problems. I think I'm banned now because of all the errors, currently I have a 23 hour backoff for downloading new WUs.

Is it only me having problems ?


No I'm having the same problems, some units work and some just error out for me. It's also happening in both Linux and Windows machines for me.
ID: 2084 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Questions and Answers : Web site : Collatz - something wrong today (2019-11-16)


©2020 Jon Sonntag; All rights reserved