Are you trying to fill all 7 slots with a single slot GPU or just the 4 PCIe 16x slots on that MB? Even with blower cards, you should not "stack" more that 2 GPUs- any middle GPU(s) will not get air. If you populate the four 16x PCIe slots of that MB, there would be enough room but you would still be better with mid-range dual slot GPUs, especially if they are blower (in my opinion).
runs very nicely with 4 Titan V's (real dual slot gpus). I accidentally bought 2 without blowers so I have a small floor fan shoehorned in there.
I am exploring populating all 7 slots with single-slot gpus. I am not running Pcie intensive tasks so the 8x slots should not make any difference.
I am exploring possible hobbyist-priced solutions to drive my system RAC beyond a possible 10M level. If I had unlimited funding I could get in line for some of those $30,000 gpus... :)
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Are you trying to fill all 7 slots with a single slot GPU or just the 4 PCIe 16x slots on that MB? Even with blower cards, you should not "stack" more that 2 GPUs- any middle GPU(s) will not get air. If you populate the four 16x PCIe slots of that MB, there would be enough room but you would still be better with mid-range dual slot GPUs, especially if they are blower (in my opinion).
runs very nicely with 4 Titan V's (real dual slot gpus). I accidentally bought 2 without blowers so I have a small floor fan shoehorned in there.
I am exploring populating all 7 slots with single-slot gpus. I am not running Pcie intensive tasks so the 8x slots should not make any difference.
I am exploring possible hobbyist-priced solutions to drive my system RAC beyond a possible 10M level. If I had unlimited funding I could get in line for some of those $30,000 gpus... :)
I have talked with PNY if the single slot a4000 GPU could be stacked like that (7x) since they make the enterprise/workstation GPUs. They said there would be no way they would cool (the middle GPUs). Single slot cooling solutions are REALLY weak (compared to dual) and when pushing these GPUs to 100%, 24/7, I just don't think it would work. If it was mixed use on and off GPU processing, then it probably would but I am not sure if would for this type of work. I could be wrong though!
Are you trying to fill all 7 slots with a single slot GPU or just the 4 PCIe 16x slots on that MB? Even with blower cards, you should not "stack" more that 2 GPUs- any middle GPU(s) will not get air. If you populate the four 16x PCIe slots of that MB, there would be enough room but you would still be better with mid-range dual slot GPUs, especially if they are blower (in my opinion).
runs very nicely with 4 Titan V's (real dual slot gpus). I accidentally bought 2 without blowers so I have a small floor fan shoehorned in there.
I am exploring populating all 7 slots with single-slot gpus. I am not running Pcie intensive tasks so the 8x slots should not make any difference.
I am exploring possible hobbyist-priced solutions to drive my system RAC beyond a possible 10M level. If I had unlimited funding I could get in line for some of those $30,000 gpus... :)
I have talked with PNY if the single slot a4000 GPU could be stacked like that (7x) since they make the enterprise/workstation GPUs. They said there would be no way they would cool (the middle GPUs). Single slot cooling solutions are REALLY weak (compared to dual) and when pushing these GPUs to 100%, 24/7, I just don't think it would work. If it was mixed use on and off GPU processing, then it probably would but I am not sure if would for this type of work. I could be wrong though!
the problem with A4000s and most single slot GPUs (models With display outputs), is that it doesn’t exhaust out the back of the case. It’s mostly exhausting out the small slots on the cooler shroud near the back of the GPU. So when you have them butted up next to each other, there’s not really anywhere for the hot air to go.
the Titan V, and most other dual slot blower cards actually have venting on the second slot, so it works better, even with them butted up to each other, since usually there’s still a small gab for airflow into the fan. This is even better if you have the cards in a high airflow case or server chassis.
plus the Titan V in particular only uses about 120-140W (same as an A4000) when crunching BRP7 or O3AS so having the dual slot cooler on it is pretty capable, even with restricted airflow.
the only single slot GPUs that would be appropriate in a 7x stacked setup would be Tesla/Instinct data center level cards without display output and good venting out the back of the case.
the problem with A4000s and most single slot GPUs (models With display outputs), is that it doesn’t exhaust out the back of the case. It’s mostly exhausting out the small slots on the cooler shroud near the back of the GPU. So when you have them butted up next to each other, there’s not really anywhere for the hot air to go.
I missed that completely. :(
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
the problem with A4000s and most single slot GPUs (models With display outputs), is that it doesn’t exhaust out the back of the case. It’s mostly exhausting out the small slots on the cooler shroud near the back of the GPU
The AMD W7600 that I run on one of my machines is actually pretty good in that department. It still exhausts some heat to the sides and a blower fan to remove that is recomended, but most of the heat gets out of the case trough the rear panel which is basically a grid with many holes. Yes it would be better without ports but it's surpriseingly good even with the ports.
Although since I work on the machine I'd recomend for GW tasks a card with more than just 8GB of RAM. For crunching only it's fine. I have a RAC of about 900.000 on that machine with some CPU tasks.
Edit: power consumption of the GPU running at 3x is ~110 watts.
Why not just hook up a mining rig instead of rebuilding everything?
The Mining Rig MB's (8 slots) are too small cpu's for what I want to do. Usually 4c sometimes 8t.
Setting up any MB with Ribbon Cables or Mining Risers have been unstable/not worked at all for me. So I swore off them.
I had that experience as well but didn't know it was 'the norm', thanks.
I am uncertain if it is 'the norm' because a number of people have been successful. I spent a lot of time, and a fair amount of money and consistently lost my patience. Unfortunately, it is easy to lose since I can only emulate patience. I don't really have any native patience.
It does look like if I really want 7-8 gpus I would have to byte the financial bullet and buy a server mb/chaises that support 8 dual slot gpus. Since I have good evidence that a Titan V (with a blower) will fit inside 2 slots.
So most likely I just will be running my current setup until I win the Lottery...
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
i think Tom had trouble properly troubleshooting what the exact issue was. probably bad risers, or bad BIOS settings, or both.
I know he previously used some "EZDIY FAB" brand risers, which are *OK* for gen3, but you really have to QC them yourself to make sure they arent bad. I use/used them and had to return/replace maybe 20-30% of them before I got all good ones. most of the no-name brand risers longer than about 30mm are junk and wont even work at gen3 speeds.
with the Asrock EPYCD8 board, that is not gen4 capable, you have to be careful when using gen4 GPUs with gen3 risers and Rome CPUs. since Rome CPUs support gen4, and when using a gen4 GPU (like rtx 30-series and up) with gen3 risers, if the BIOS is set to auto for link speed, there are no limits in the BIOS to prevent the GPU from trying to run gen4. so you need to make sure the slots are set to gen3 explicitly in the BIOS in this case. Tom shouldnt have that issue with Titan V GPUs though since they will never try to run gen4, as they are limited at the GPU itself. this is technically a BUG in BIOS logic in my opinion, but the easy workaround is to just set all slots to gen3 speeds.
risers are no problem when you set it up correctly. my 8x Titan V system is running the sketchy EZDIY risers, but I QC'd them to ensure they all work, and they aren't that long. most are like 20cm. my 8x V100 are also on risers, though a different kind of setup.
linkup makes good cables from what I saw. but they are more expensive per cable. especially the gen4 rated ones.
I would encourage Tom to try them out, and just eat the cost. it'll be cheaper than a whole new platform. and a lot quieter.
I would encourage Tom to try them out, and just eat the cost. it'll be cheaper than a whole new platform. and a lot quieter.
I love "a lot quieter". :)
Found an Nvidia M4000 8 GB for $117 so I ordered it. I am "curious" like George is.
I have kept my mining rig frames so jumping back up to a riser-type approach is not very hard if I can get past my troubleshooting issues.
More to ponder for my Christmas presents as I pay down some of my revolving credit :)
I wonder if I could get bifurcating to work if I could mix and match, rtx 3080 ti's (2), Titan V's (2, sell non-fan ones), 10 Nvidia M4000's. :)
That would give me 14 x 3 threads or a 42 thread rig. Probably shouldn't though :) Most likely something would break.
:(
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Boca Raton Community HS
)
https://einsteinathome.org/host/13190241
runs very nicely with 4 Titan V's (real dual slot gpus). I accidentally bought 2 without blowers so I have a small floor fan shoehorned in there.
I am exploring populating all 7 slots with single-slot gpus. I am not running Pcie intensive tasks so the 8x slots should not make any difference.
I am exploring possible hobbyist-priced solutions to drive my system RAC beyond a possible 10M level. If I had unlimited funding I could get in line for some of those $30,000 gpus... :)
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Tom M wrote: Boca Raton
)
I have talked with PNY if the single slot a4000 GPU could be stacked like that (7x) since they make the enterprise/workstation GPUs. They said there would be no way they would cool (the middle GPUs). Single slot cooling solutions are REALLY weak (compared to dual) and when pushing these GPUs to 100%, 24/7, I just don't think it would work. If it was mixed use on and off GPU processing, then it probably would but I am not sure if would for this type of work. I could be wrong though!
Boca Raton Community HS
)
the problem with A4000s and most single slot GPUs (models With display outputs), is that it doesn’t exhaust out the back of the case. It’s mostly exhausting out the small slots on the cooler shroud near the back of the GPU. So when you have them butted up next to each other, there’s not really anywhere for the hot air to go.
the Titan V, and most other dual slot blower cards actually have venting on the second slot, so it works better, even with them butted up to each other, since usually there’s still a small gab for airflow into the fan. This is even better if you have the cards in a high airflow case or server chassis.
plus the Titan V in particular only uses about 120-140W (same as an A4000) when crunching BRP7 or O3AS so having the dual slot cooler on it is pretty capable, even with restricted airflow.
the only single slot GPUs that would be appropriate in a 7x stacked setup would be Tesla/Instinct data center level cards without display output and good venting out the back of the case.
_________________________________________________________________________
Ian&Steve C. wrote: the
)
I missed that completely. :(
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Ian&Steve C. wrote:the
)
The AMD W7600 that I run on one of my machines is actually pretty good in that department. It still exhausts some heat to the sides and a blower fan to remove that is recomended, but most of the heat gets out of the case trough the rear panel which is basically a grid with many holes. Yes it would be better without ports but it's surpriseingly good even with the ports.
Although since I work on the machine I'd recomend for GW tasks a card with more than just 8GB of RAM. For crunching only it's fine. I have a RAC of about 900.000 on that machine with some CPU tasks.
Edit: power consumption of the GPU running at 3x is ~110 watts.
Tom M wrote: mikey
)
I had that experience as well but didn't know it was 'the norm', thanks.
mikey wrote: Tom M
)
I am uncertain if it is 'the norm' because a number of people have been successful. I spent a lot of time, and a fair amount of money and consistently lost my patience. Unfortunately, it is easy to lose since I can only emulate patience. I don't really have any native patience.
It does look like if I really want 7-8 gpus I would have to byte the financial bullet and buy a server mb/chaises that support 8 dual slot gpus. Since I have good evidence that a Titan V (with a blower) will fit inside 2 slots.
So most likely I just will be running my current setup until I win the Lottery...
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
ASUS WS C621E works excellent
)
ASUS WS C621E works excellent with 7 GPUs using riser cables from Linkup.
Great board ...
BTW: All my PCs are running with such cables - never ever had problems ... (except that they are way to expensive - but worth it)
keep up the good work ...
[EDIT]:
Linkup from 30cm to 75 cm PCIE4.0 + no BIOS fiddling ...
Most of them are coupled to about +150cm - no problem
and I don't worry about the speeds ...
Life should stay/be easy - stay cool !
i think Tom had trouble
)
i think Tom had trouble properly troubleshooting what the exact issue was. probably bad risers, or bad BIOS settings, or both.
I know he previously used some "EZDIY FAB" brand risers, which are *OK* for gen3, but you really have to QC them yourself to make sure they arent bad. I use/used them and had to return/replace maybe 20-30% of them before I got all good ones. most of the no-name brand risers longer than about 30mm are junk and wont even work at gen3 speeds.
with the Asrock EPYCD8 board, that is not gen4 capable, you have to be careful when using gen4 GPUs with gen3 risers and Rome CPUs. since Rome CPUs support gen4, and when using a gen4 GPU (like rtx 30-series and up) with gen3 risers, if the BIOS is set to auto for link speed, there are no limits in the BIOS to prevent the GPU from trying to run gen4. so you need to make sure the slots are set to gen3 explicitly in the BIOS in this case. Tom shouldnt have that issue with Titan V GPUs though since they will never try to run gen4, as they are limited at the GPU itself. this is technically a BUG in BIOS logic in my opinion, but the easy workaround is to just set all slots to gen3 speeds.
risers are no problem when you set it up correctly. my 8x Titan V system is running the sketchy EZDIY risers, but I QC'd them to ensure they all work, and they aren't that long. most are like 20cm. my 8x V100 are also on risers, though a different kind of setup.
linkup makes good cables from what I saw. but they are more expensive per cable. especially the gen4 rated ones.
I would encourage Tom to try them out, and just eat the cost. it'll be cheaper than a whole new platform. and a lot quieter.
_________________________________________________________________________
Ian&Steve C. wrote:I would
)
I love "a lot quieter". :)
Found an Nvidia M4000 8 GB for $117 so I ordered it. I am "curious" like George is.
I have kept my mining rig frames so jumping back up to a riser-type approach is not very hard if I can get past my troubleshooting issues.
More to ponder for my Christmas presents as I pay down some of my revolving credit :)
I wonder if I could get bifurcating to work if I could mix and match, rtx 3080 ti's (2), Titan V's (2, sell non-fan ones), 10 Nvidia M4000's. :)
That would give me 14 x 3 threads or a 42 thread rig. Probably shouldn't though :) Most likely something would break.
:(
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!