There are many different system configurations out there. Many of them are unique in many ways. To be honest with you, the question I get asked the most is “How many plots can I run and what settings should I use.” Usually accompanied with the person’s machine specs. In this post, I will guide you in figuring out the plotting configuration that will make the most out of your machine. Chia touches almost everything, so we’re going to cover basically all of it. It may get lengthy, but you should be able to figure out your own system once we’re done.
There are three components that will determine how many plots you can have in parallel; CPU, RAM, Temp Disk Size. First step is to figure out where the limiting factor is on your system. Use these formulas below:
- CPU is limited by this formula: (Cores + Threads)/2
- RAM is limited by this formula: (Total System RAM in MB)/3400MB. Round this down.
- Temp Space is limited by this formula: (Total Temp Space)GB/250GB. Round this down.
Let me explain each formula and its purpose, starting with the first one, CPU. This formula is your cores plus your threads divided by two. This is done because of the way Chia does its plotting. Chia plots in four phases. Phase 1 and 3 usually take the longest. When you set your threads for the plotter, that only affects Phase 1. Phase 2, 3 and 4 are all single threaded. So, as plots move out of Phase 1, it frees up an extra thread which you can then use for a new plotter. CPU can be oversubscribed a little bit meaning that you can go over your total thread count, it will just slow down a bit. It will not crash the plotters.
RAM is pretty straight forward. Normally when using two threads, the optimal amount of RAM is 3389. In order to make the math a bit simpler, I use 3400. This is the amount that each plotter will use at some point of their plotting process. The reason to round down here is that RAM cannot be over allocated like the CPU. If you run out of RAM, its going to cause the plotters to error.
Temp space is also straight forward. Each plotter will use 256GB (Edit: 250GB now) (aka 232GiB) as temp space. The reason to round down here is because over allocating temp space is a bit difficult. You can do it, I’ve seen it done, 9-10 plotters on a 2TB NVMe. The key is the delay between the plotters. That value can only be figured out with trial and error of your own system.
Let’s proceed and use my system as the example.
- My CPU is a 5900X which is 12 core/24 thread. Following the formula, 36/2, I come up with 18.
- My system has 32GB of RAM. Following the formula, 32000/3400, I come up with 9.4. Rounding down, its just 9.
- I have two 2TB NVMes. Following the formula, 4000/250, I come up with 16. Rounding down, its still 16.
Alright, I have the numbers for my system. The limiting factor in my system is the RAM. I can only run up to 9 plotters at one time with the optimum amount of ram. When using this formula, you will use 2 threads and 3400 RAM as the plotter settings. This will give you a good starting point to your system out of the box. The next thing to figure out is the Delay (also known as stagger). The delay depends on its own set of questions:
- Are all plotters writing the final plot to the same HDD? If so, you must stagger at least 30 minutes between plotters. It will take 20 minutes to write a plot to an HDD. This is so that two plotters do not try to write to the same HDD at the same time. It will cause a huge bottleneck.
- Do you have multiple final HDDs? If you do, good, you can start groups of plotters at the same time instead of just one. For example, if you have 3 final external HDDs, You can start the plotters in groups of 3 (each with a different destination drive), stagger by 30 minutes, then start the next three. Be careful here however, Make sure you have enough resources to start each group of plotters. Your stagger may need to increase to an hour or more if you don’t have enough CPU threads available.
- Do you have an NVMe or SSD as the final drive? If you do, great. You can launch plotters with a 5 minute stagger since the copy time is so short. Some people do this in order to finish plots as fast as possible. This allows the plotters to continue on the next plot while a script transfers the plots to an external HDD. This one takes a bit more trial and error to figure out how many you plotters you can start at once.
Now, there is one last part. Plot speed is partially depended on the type of Temp Space you have. NVMe is the best, SAS and SATA SSD is next, HDD is the worst. This next bit of information has to do with the NVMe Brand. Yes, the make and model of the NVMe you are using. Most Chia people are aware that NVMe endurance is important, TBW (TeraBytes Written), when plotting for Chia. What people have not been aware of is the Sustained Write Performance of the NVMe Drive. This normally isn’t in the specs of an NVMe because consumer NVMes aren’t really used in the manner that Chia uses them. Some NVMe drives feature an SLC Cache that provides the high performance numbers printed on the box. In many workloads, this is fine. For Chia, this is not a good feature. The NVMe drives that have this cache lose a lot of performance once the SLC Cache is full. This all boils down to the controller on the NVMe. Some NVMe’s have a good controller and some don’t. Take for example, the Adata XPG SX8200 Pro 2TB. It has great performance out of the box, but look at its sustained write performance (Credit to Tom’s Hardware):
In the graph, the x-axis is the total amount of data written. The Adata is the red line in the graph. You can see that after about 120GB of data written, the performance drops significantly. The same is true for many others in that graph. The Patriot VP4100 goes from 4000+ MBps to 500 MBps after 175GBs written. This is why some people are confused that they are not getting the performance they should be getting. Or why the first round of plots were much faster than the second round. The NVMe’s with this feature will not provide the best performance with the way Chia works. Each plotter writes 1.4TB to generate just one plot. If you have many in parallel, this cache is filled very quickly and your phase one could take hours and hours to finish.
With this information, you should now be able to have a good starting point. Now when you go to buy an NVMe, there are three things you must consider; Capacity, Endurance, Sustained Write Performance.
I would like to make you aware of something else as well. There is a lot of old data out there from previous versions of the plotter that will also trip new people up. This isn’t on purpose, its just that the Chia Client is being updated so frequently that its hard to update every piece of old information. Usually people copy and paste commands without knowing what they do, so here is a few things to look out for:
- Using 3400 RAM for 2 threads in a plotter setting is close to ideal. Setting more RAM is a waste of the resource if you can have more in parallel. I believe on the Chia team’s wiki it says that 6750 RAM is the max for bitfield. This is not entirely the case anymore. Increasing the RAM does provide a minimal speed boost (1-2%), but it is not worth it if another plotter can run.
- Some scripts have the “-e” flag in the command for the plotter. This is how it used to work for a higher plot speed, but not any more. Remove “-e”. Do not use “-e” in any commands unless the computer is really really old. The “-e” will make the plotter use 356GB of temp space instead of 250GB. Also it is slower. This has been true since version 1.0.4 of the Chia Client.
- Some scripts have the “-2” flag in the command for the plotter. This is not needed initially. Remove this because it is unnecessary. If you want to experiment with it, then do it later when you have experience. In my testing, RAID NVMe’s were faster than using “-2” anyways.
- Using more than 4 threads isn’t that beneficial when running plotters in parallel. Some people like to run with 8 threads or 16 threads. No. This will actually hurt performance. In my testing there was only a 5 minute difference between 4 threads and 6 threads in plot speed. Not worth it. Use 2 or 4 when possible. Remember, if you use 4 threads, you must use at least 3408 RAM.
- Have caution using really big NVMe drives (more than 2TB). For example, having a 4TB NVMe with only a 4x PCIe bus will get saturated quickly if you try to run 15-16 plotters on it at the same time.
Keep on plotting on!
A great write-up again Alex,
Wondering if you can elaborate a bit more on how you came up with the rule of the thumb you use for calculating CPU bottleneck. Using your example lets say you actually had 128GB RAM. This time the limit would be the Drives which would limit you down to 15.
As per your previous article a good idea would be to use 2-3 threads per plot. Let’s say we use 2 for this example.
Then that would mean that in total you would be using 30 threads on your CPU with 24 threads.
So questions now start to arise as to how thread allocation works in Chia.
I suppose the simple answer would be that under-utilization of threads from plots in phase 3 or 4 would suffice for the overlap.
However this only brings on more questions regarding how the thread allocation works!
Hey again! You are correct that I am missing the details on my theories. I will update the post to better explain whats going on here.
You obv didnt read well enough. He stated that you are using 4 phases in total. 3 phases are using one thread while the first one is using 2.
That means that you can either do the math as he suggested 12c+24t = 36/2 = 18.
Or you can simply use logic and devide the total of threads with 4 (since 4 phases) and then just deduct the result with the total count of threads, following like this;
24/4=6, 24-6=18.
You basically deduct 4 from 24 becuase first phases uses two threads instead of one.
As for your Q.
With 128 gigs of RAM you’d have 37.6 or 38 lanes open for parallel operations.
That beings said the limiting factor here would be the disk space and CPU. with CPU provinding 18 lanes and disk 15
which means that you would be operating with the speed of the weakest link, meaning max operations per time would be 15.
If you doubled the volume of disk then the limiting component would become CPU.
“You obv didn’t read well enough”…”do the math as he suggested”…Dude. Get out of the basement and work on your social skills. There’s no need to shame people for asking questions. It’s how learning happens. And it doesn’t make you look smart. It makes you look rude and poorly socialized. Happy plotting, friend!
I wasnt trying to be rude or anything, just re read twice to be sure next time. I’ve ton of people misreading something or not even thinking through what they just read and saying things like theres nothing there or yari yari yada… just reread whole thing slowly and with questions in mind 😀
Then ask, fair enough? 😀
Hi Alex, thank you very much, very helpful as always.
The only thing you should do now is, pls start a chia-pool and let us plot/farm together.
regards from germany
thank you again Alex for the valuable information.
I am running the same exact system specs as you. so its much easier to follow your steps. My issue so far is that it takes 10.5 hours to complete the 12 plots in parallel , i was hoping to have it faster then this. I used your script to run with delayed starting time 6 plots to each tmp drive that copies after to a different HDD.
Or to build another faster system, what would it take. ? 64 Cores, 256GB ram, 4 M2 or maybe U2.
If you can make the system have 64GB of RAM that will speed things up because then you can use the right amount of RAM per plotter.
Alex thanks again for this blog
the sustained write speed was educational and by pure luck my NVMe selection fell on the correct corner on this.
1. Assuming there is no problem on copying final file to -D destination and therefore, there is no need for stagger, (1a) Then in the above config we can run all 9 plotters at the same time – correct? (1b) are all the plotters writing to the 2TB NVMe at the same time? (1c) or do we need to make a subdirectory in the NVMe ?
2. I noticed on temp size – it is indicated 256GB for calculation. The following is what I am seeing at the end of every plot creation – “..(Temp) space excluding final file size 269GiB”. I assumed that including final PLOT file size of 101.35 was what was required if I did not have -2. In other words where there is no -2 specified then -t should have at least around 372 GiB. Please can you clarify
3. after phase 1 I thought it will be r-1 threads become free . That explains the “one thread become free” after Phase 1..since r=2 – right?.
Hi gigifarmington, thanks for the comment. 1) Even with an NVMe as a destination drive, it takes 2 minutes to copy the plot over. I would say stagger 5 minutes. You can write all of them to the NVMe, you don’t need to make subdirectories. 2)Yes, this is a display bug in the plotter. I have verified that it is 256GB. If you run without bitfield it will be 356GB. 3)This is right if you did 2 threads. If you do 4 threads, 3 threads become free after phase 1. It will always go down to 1 thread after phase 1.
Hi Alex
thank for you blog . i curious about “Using 3400 RAM for 2 threads in a plotter setting is close to ideal.” does this mean per plot right? 2 thread mean CPU thread?
Hi Tony, Yes per plotter. And 2 thread means 2 CPU thread.
Do you use any plot manager from Github? or your plotting purely done by CLI?
With i9 9900k, 16Thread, 64GB, 2.5TB SSD in total, what would be your recommended setup?
I’m interested in your CLI config as well if you dont mind sharing 🙂
Hi roph, I don’t have a plot manager, but now that more time has passed maybe there is a good one for Windows. You can find the script I use in this blog post: https://thechiafarmer.com/2021/04/21/optimizing-plotters-in-windows/
Hello great article.
I wonder what do you think about making temp disk out of RAM?
In my case, I have 128GB RAM DDR3, 4TB (2×2) NVMe SSD in RAID 0 and 2 CPU Xenon E5-2450L
CPU and space for temp is a limitation for plotting in my case. Now I am testing 15 parallel plotters with 20 min delay, 2 threads per plot and 7GB RAM.
Works on GUI and UBUNTU. Do you have any advice?
Hello dkaro, you have nvme’s so a RAM disk won’t help as much. I think you can plot 17 in parallel, but it will take trial and error to figure out the right delay.
What do you think;
600TB of space
TR 3990x 64c
320GB RAM
96 for CPU
94 for RAM
Tho does the config of disk matter, like 8TB*75disks or 16TB*37.5 disks
does the number of disks matter?
otherwise 2343 lanes from HDD? but that would be within 31 cycles in total per 8TB disk
or 62x per 16TB
Tho if i had 75 disks per 8TB, would that mean that every 30min each disk would be filled for 256GB basically using 1 lane per disk or 75 lanes per 30min in total?
Hi Klemn, Just need to ask some questions. Is that 600TB temp space? or final storage space? 2)What do you mean when you say lanes? PCIe lanes? I don’t see NVMes anywhere or SSDs. Are you planning to use HDDs as temp space? HDDs can only support 1 plotter using it as temp space at a time.
Hi, 600TB as final storage space. I’d have 4-8TB of NVME 980 pro at the end. in raid probably.
And ye, NVME would be temp space as i understood correctly.
But let’s say I had 600TB of final storage, does count of physical disks matter?
And I came up with my own calculation now i dont know if it is right or not so here pass your judgement on it 🙂
If i have 4TB of NVMe, R/W speeds are abount 7000MB/s, having 600TB of storage
I will be doing 15.6 plots in total per 4.7min?
4TB / 256GB = 15.6 plots
256GB / 7GB (NVME SPEED) = 36.5s per plot
Since we have 2 NVMEs we can do the math like this (15.6 plots * 36.5s) / 2 = 285s = 4min 45s.
——
Even if this isnt correct I’d like to know a number of days or hours it takes to fill the 600TB storage with 4TB of NVME. If the physical count matters then let’s say I’d have around 33 physical disks by 18TB~
Hello! Could you explain, why we can sum physical and logical CPU cores in our formula? I thought that threads are simply physical cores x2, so they’re the same thing (1 core = 2 threads), but for different use scenarios. How can we sum them up in calculations? Please expllain this moment a little bit, because I used to divide only number of threads, because I thought that they’re already cpu cores and nothing extra. Thank you!
Hi again, You may have read an old version of my post. I put the explanation in the post now.
Oh, now I get it, thank you! So basically this formula doesn’t mean that cpu uses both Cores AND Threads at the same time (that what messed me a little bit). So it’s just simple approach to get approximate number of plots based on fact that each plot unchains all threads and keeps only 1 for itself.
For instance if I have 8 core 16 threads cpu, based on formula with 4 threads per plot I will be able to plot (8+16)/4 = 6 plots in parallel. For instance if I plot to 2HDDs, then 4 of plots will be at P1 (4 thread each), 1 at P2, 1 at P3, 0 at P4. Then plots*threads will be: 4*4+1*1+1*1+0*1 = 18 threads which is more than CPU has.
Does that mean that I need to lower number of threads per plot, or somehow stagger them more to avoid 4 plots in Phase 1? Or if I want to run 8 plots in parallel, then I have to use 2 threads per plot..will it give more advantages vs 6 plots/4 threads with a little bit cpu-overload.
Have you tested how much “overloading” your cpu with extra threads affecting plotting times?
Thanks again! 🙂
Hello Alex, thank you, this helped me much as a beginner.
I have some Questions, sorry for my english.
I have an old Xeon e3 1231v3 (4+8), 16 Gb ram and one M2 SSD 970 Evo Plus 1TB, connected with an Adapter to my PCI Express, because my old Mobo does not have a M2 slot. I also got one Crucial MX 500 SSD. One TOshiba external 1 TB HDD. My calculations say my bottleneck is the ram (4,7= 4).
As this is my old System i think i got good components to start chia farming.
Iam a absolute beginner and i want to know if i made any mistakes and do i need anything to buy?
I know its not a pro setup, but i think it will help me to get into this, but sadly i got almost no Money to invest more, s it would be really nice if it works with what i already have.
thank you Alex and happy Farming
Hi Markus, this is a good system to start with. The only issue I see is that you will need storage to store all the plots. The 1TB external can hold 9 plots and the 500 MX SSD can hold 4. You can farm these and hopefully be lucky to win 2XCH. It is a really long shot though. If you do win, you can sell those to get more equipment.
Thanks for the answer. Well i did some changes, sold my 3D printer and now iam going to buy a i9 10850k with 64gb Ram and two 2tb NVME’s. i think it will pay out and i really hope i can make money from it in time. You post really helped me to figure out how this works 🙂
Does that mean if i had 600TB final storage as destination, would that house in 5555 plots in total?
Where does CPU and NVME then come in? If i am able to to 96 in parallel with CPU, does that mean CPU can plot 96 plots per day? or it depends?
Why your NVME capacity is only 2TB and the formula: (Total Temp Space)GB/256GB, but you said that:
“I have two 2TB NVMes. Following the formula, 4000/256”.
I found this website (https://plot-plan.chia.foxypool.io/) to plan my plotting, after input 2TB of Temp Space, it also said that I can plot 18 plot each time, how to do that ?
Hi Van, that website tells you how many finished plots you can put in a destination drive. It’s not for the temp drive.
Thanks, I got it, but why you said 4000/256 ?
Because I have two 2TB drives for temp space. It’s 4000GB available divided by 256.
Thanks for your answer ! I have another question, what is the best time to start stagger ? (which phase), and Is stagger mean start all next plot job or not (less than) ?
Stagger means the time between the plotters. You need to look at your destination drive and computer resources to determine how much delay to put. Its unique for every system.
Your phase 1 is about 2400second?
No, it’s more like 10000ish seconds. Are you talking about something else?
plotter starts every 40 minutes how many at one moment do you have a plot in phase 1?
I have a 64 thread, 256GB mem server.
should my limiting factor be the cpu threads or memory?
probably how fast you can make plots and plot them prob NVME.Gen4 is the best.
Its a cloud server so cant have the same NVMe bandwidth as the real one! But I need to set the maximum number of parallel plotters to have max plots/day.
Hi Alex. I cannot understand how many threads and maximum RAM should I specify when creating three sections in parallel?
I have intel i7 8700, RAM 32GB 2666 Mhz, ssd Samsung 1TB 870 EVO
Alex,
Are modern systems like a Threadripper, limited by PCIE/Motherboard bandwidth? Take for example, you had a 32 parallel plotting system and assuming you have enough resources for everything (CPU, RAM) and for temp SSD space if you plot across 8 nvme drives, 4 allocated to each, is that too much I/O for the bus to handle?
Hi Alex,
I have the exact same system as you have.
In your old post (https://thechiafarmer.com/2021/04/21/optimizing-plotters-in-windows/) you suggest using 2400 ram to have 12 plots in parallel.
What was the difference between 2400/12 and 3400/9?
Hi Rayan, this is interesting. What are your plot times with 3400/9?
I’m now benchmarking 3400/9 (will post my results tomorrow)
my 2400/12 results:
mean max min
PHASE_1 05:00:57 05:59:47 03:37:02
PHASE_2 01:59:11 02:18:39 01:48:44
PHASE_3 04:23:52 04:41:24 04:06:54
PHASE_4 00:32:56 00:36:16 00:27:12
COPY_TIME 00:06:38 00:10:38 00:04:10
TOTAL_TIME 11:56:59 13:30:41 10:14:32
https://i.ibb.co/PjTY7dQ/Untitled.png
The reason I do 12 plotters is because its about 8 hours per 12 plots. That means 36 plots a day or 3.6TiB/day. With 8 plotters, I get 8 every 6.5 hours. This means a little less than 32 plots a day or 3.2TiB/day. 12 produces more plots per day than 8 for me.
in 3400/9 I’ve got 26 plot per day.
I don’t really have final results in csv (but this is what I see in swar-plot-manager)
what is your stagger settings for 2400/12? anything special?
For the 2400/12, right now I’m doing 2 every 40 minutes. It gives 36 plots a day. 3400/8 give me 32 plots a day.
you already have better performance than me even in 3400/9
let’s compare our hardware spec
this is my system spec:
CPU: Ryzen 9 5900x
RAM: 32GB GSkill (F4-3600C18D-32GTZN)
Tmp: 2x2TB Transcend TS2TMTE220S 2TB
Dst (buffer): Samsung 870 EVO 1TB
robocopy to this:
Final dst: Western Digital 12TB My Book New WDBBGB0120HBK
That is a bit odd. My hardware spec is on my About page at the top right. https://thechiafarmer.com/about/
chia is just like a mill that grinds money. For better hardware, you need to constantly throw tokens into the system. there is no end to it. Finally, you find yourself setting up a huge data center at home. Perhaps the most sensible thing to do, hang out with the players available hardware.
Hi Alex, I have a question to optimize my plotting, PC;
Intel Core i5-11400 6 cores, 12 threads (9 plots)
DDR4 2x16GB G.Skill 3200MHz (9,4 plots)
Adata SSD 2TB XPG Gammix S50 Lite, PCIe M.2 (8 plots – bottleneck on this component)
3x WD 10TB 3.5″ 7200rpm, 256MB cache, SATA
So I did 3 groups of parallel plots each group having to do 2 plots. Since I have the starting area M2 SSD and 3 HDDs as final destination. Added each group to 1 HDD.
I gave them all 4000MB ram, 2 threads and delay of 150min for the next plot to start working. And I added each group every 30min. So I am hoping not to overwhelm the cpu like the first day I tried.
I added all of the 3 groups the same moment with those settings and the delay on parallel plots was 30min, man was it horrible. Took me around 12-13h to do those 6 plots.
So I just want to ask you, what setting or delay time would you suggest me adding or reducing to get more plots per day? Any way I can exactly calculate all of those settings for my PC. I check some guides that talked about the phases done and then how to calculate delays etc. But it’s a bit confusing for me. Thanks for any advice!
ryzen 9 9500 7 mhz 24 processor
32 coaches
samsung 980 pro 2tn nwme m 2
How do I write with the fastest and most convenient way.
Please could you give a configuration example for 4 plots, assuming we have the necessary requirements. Example of memory and threads suitable for 4 plots in parallel.
Hi alex, sorry for this dumb question but i see in your article the total amount of plots in parallel right?, but what is the mat to calculate amount of plots per day.
Hi Miguel, The way you figure plots per day is you see who long it takes to make the plots in parallel, Then you divide 24hrs by that number. This will give you how many plots are generated per plotter. Then multiply it by the number of plotters you have. That should give you the total plots per day.