There are many different system configurations out there. Many of them are unique in many ways. To be honest with you, the question I get asked the most is “How many plots can I run and what settings should I use.” Usually accompanied with the person’s machine specs. In this post, I will guide you in figuring out the plotting configuration that will make the most out of your machine. Chia touches almost everything, so we’re going to cover basically all of it. It may get lengthy, but you should be able to figure out your own system once we’re done.
There are three components that will determine how many plots you can have in parallel; CPU, RAM, Temp Disk Size. First step is to figure out where the limiting factor is on your system. Use these formulas below:
- CPU is limited by this formula: (Cores + Threads)/2
- RAM is limited by this formula: (Total System RAM in MB)/3400MB. Round this down.
- Temp Space is limited by this formula: (Total Temp Space)GB/250GB. Round this down.
Let me explain each formula and its purpose, starting with the first one, CPU. This formula is your cores plus your threads divided by two. This is done because of the way Chia does its plotting. Chia plots in four phases. Phase 1 and 3 usually take the longest. When you set your threads for the plotter, that only affects Phase 1. Phase 2, 3 and 4 are all single threaded. So, as plots move out of Phase 1, it frees up an extra thread which you can then use for a new plotter. CPU can be oversubscribed a little bit meaning that you can go over your total thread count, it will just slow down a bit. It will not crash the plotters.
RAM is pretty straight forward. Normally when using two threads, the optimal amount of RAM is 3389. In order to make the math a bit simpler, I use 3400. This is the amount that each plotter will use at some point of their plotting process. The reason to round down here is that RAM cannot be over allocated like the CPU. If you run out of RAM, its going to cause the plotters to error.
Temp space is also straight forward. Each plotter will use
256GB (Edit: 250GB now) (aka 232GiB) as temp space. The reason to round down here is because over allocating temp space is a bit difficult. You can do it, I’ve seen it done, 9-10 plotters on a 2TB NVMe. The key is the delay between the plotters. That value can only be figured out with trial and error of your own system.
Let’s proceed and use my system as the example.
- My CPU is a 5900X which is 12 core/24 thread. Following the formula, 36/2, I come up with 18.
- My system has 32GB of RAM. Following the formula, 32000/3400, I come up with 9.4. Rounding down, its just 9.
- I have two 2TB NVMes. Following the formula, 4000/250, I come up with 16. Rounding down, its still 16.
Alright, I have the numbers for my system. The limiting factor in my system is the RAM. I can only run up to 9 plotters at one time with the optimum amount of ram. When using this formula, you will use 2 threads and 3400 RAM as the plotter settings. This will give you a good starting point to your system out of the box. The next thing to figure out is the Delay (also known as stagger). The delay depends on its own set of questions:
- Are all plotters writing the final plot to the same HDD? If so, you must stagger at least 30 minutes between plotters. It will take 20 minutes to write a plot to an HDD. This is so that two plotters do not try to write to the same HDD at the same time. It will cause a huge bottleneck.
- Do you have multiple final HDDs? If you do, good, you can start groups of plotters at the same time instead of just one. For example, if you have 3 final external HDDs, You can start the plotters in groups of 3 (each with a different destination drive), stagger by 30 minutes, then start the next three. Be careful here however, Make sure you have enough resources to start each group of plotters. Your stagger may need to increase to an hour or more if you don’t have enough CPU threads available.
- Do you have an NVMe or SSD as the final drive? If you do, great. You can launch plotters with a 5 minute stagger since the copy time is so short. Some people do this in order to finish plots as fast as possible. This allows the plotters to continue on the next plot while a script transfers the plots to an external HDD. This one takes a bit more trial and error to figure out how many you plotters you can start at once.
Now, there is one last part. Plot speed is partially depended on the type of Temp Space you have. NVMe is the best, SAS and SATA SSD is next, HDD is the worst. This next bit of information has to do with the NVMe Brand. Yes, the make and model of the NVMe you are using. Most Chia people are aware that NVMe endurance is important, TBW (TeraBytes Written), when plotting for Chia. What people have not been aware of is the Sustained Write Performance of the NVMe Drive. This normally isn’t in the specs of an NVMe because consumer NVMes aren’t really used in the manner that Chia uses them. Some NVMe drives feature an SLC Cache that provides the high performance numbers printed on the box. In many workloads, this is fine. For Chia, this is not a good feature. The NVMe drives that have this cache lose a lot of performance once the SLC Cache is full. This all boils down to the controller on the NVMe. Some NVMe’s have a good controller and some don’t. Take for example, the Adata XPG SX8200 Pro 2TB. It has great performance out of the box, but look at its sustained write performance (Credit to Tom’s Hardware):
In the graph, the x-axis is the total amount of data written. The Adata is the red line in the graph. You can see that after about 120GB of data written, the performance drops significantly. The same is true for many others in that graph. The Patriot VP4100 goes from 4000+ MBps to 500 MBps after 175GBs written. This is why some people are confused that they are not getting the performance they should be getting. Or why the first round of plots were much faster than the second round. The NVMe’s with this feature will not provide the best performance with the way Chia works. Each plotter writes 1.4TB to generate just one plot. If you have many in parallel, this cache is filled very quickly and your phase one could take hours and hours to finish.
With this information, you should now be able to have a good starting point. Now when you go to buy an NVMe, there are three things you must consider; Capacity, Endurance, Sustained Write Performance.
I would like to make you aware of something else as well. There is a lot of old data out there from previous versions of the plotter that will also trip new people up. This isn’t on purpose, its just that the Chia Client is being updated so frequently that its hard to update every piece of old information. Usually people copy and paste commands without knowing what they do, so here is a few things to look out for:
- Using 3400 RAM for 2 threads in a plotter setting is close to ideal. Setting more RAM is a waste of the resource if you can have more in parallel. I believe on the Chia team’s wiki it says that 6750 RAM is the max for bitfield. This is not entirely the case anymore. Increasing the RAM does provide a minimal speed boost (1-2%), but it is not worth it if another plotter can run.
- Some scripts have the “-e” flag in the command for the plotter. This is how it used to work for a higher plot speed, but not any more. Remove “-e”. Do not use “-e” in any commands unless the computer is really really old. The “-e” will make the plotter use 356GB of temp space instead of 250GB. Also it is slower. This has been true since version 1.0.4 of the Chia Client.
- Some scripts have the “-2” flag in the command for the plotter. This is not needed initially. Remove this because it is unnecessary. If you want to experiment with it, then do it later when you have experience. In my testing, RAID NVMe’s were faster than using “-2” anyways.
- Using more than 4 threads isn’t that beneficial when running plotters in parallel. Some people like to run with 8 threads or 16 threads. No. This will actually hurt performance. In my testing there was only a 5 minute difference between 4 threads and 6 threads in plot speed. Not worth it. Use 2 or 4 when possible. Remember, if you use 4 threads, you must use at least 3408 RAM.
- Have caution using really big NVMe drives (more than 2TB). For example, having a 4TB NVMe with only a 4x PCIe bus will get saturated quickly if you try to run 15-16 plotters on it at the same time.
Keep on plotting on!