tag:blogger.com,1999:blog-2935119175410671378.post3741785345921331290..comments2023-03-22T03:23:08.549-06:00Comments on High Performance Linux: Studying Intel TSX PerformanceAlexander Krizhanovskyhttp://www.blogger.com/profile/00939006050444455233noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-2935119175410671378.post-14865301575972195342019-10-15T00:09:32.603-06:002019-10-15T00:09:32.603-06:00Is figure 2 correct with that many cache lines ......Is figure 2 correct with that many cache lines ... shouldn't it be the same scale as figure 1?Fred123232https://www.blogger.com/profile/14409995783737492136noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-24674199526362316722015-11-12T13:49:20.958-07:002015-11-12T13:49:20.958-07:00In that test there was no data dependency while sp...In that test there was no data dependency while spinlock still synchronizes transactions on both the cores, so it has sense to expect nearly double performance gain on updating independent data on two cores in parallel.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-17852853315790180012015-11-12T07:09:12.898-07:002015-11-12T07:09:12.898-07:00pthread_mutex_trylock is not a right way to read t...pthread_mutex_trylock is not a right way to read the status of the lock. It always writes to the lock creating a conflict that abort transactions.PThreads library does not provide any method to read-(only) the status of the lock. As Alexander stated, newer glibc can support TSX lock elision out of the box. Intel TBB supports speculative locks based on TSX.Romanhttps://www.blogger.com/profile/11885595303913940411noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-59107630829329868262015-11-12T06:58:25.197-07:002015-11-12T06:58:25.197-07:00>This results shows that TSX performs 3 times b...>This results shows that TSX performs 3 times better (401ms vs 1329ms for trx_count = 1) on small transaction.<br /><br />> I expected that TSX should show much better results for the test due to more parallelism, but it isn't so...<br /><br />what parallel speedup would you expect on a two core machine?Romanhttps://www.blogger.com/profile/11885595303913940411noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-19119373932376523542014-11-15T15:35:30.489-07:002014-11-15T15:35:30.489-07:00Hi Giorgio,
sorry for the delay - there were too ...Hi Giorgio,<br /><br />sorry for the delay - there were too many business trips for the last time.<br /><br />I have few point about the program and what's possible to try:<br /><br />1. it's not clear at which point exactly transactions abort. Does it mean that you just can't start a transaction? How many CPU cores are executing the code concurrently? Is there CPU binding enabled?<br /><br />2. you use pthread_mutex_trylock() inside the transaction and the call in fact uses TSX since glib 2.18. TSX should be ok with nested transactions, but I'd try to debug the TSX code from as simple transaction code as possible and increase the code complexity until you get the code point which leads to enormous aborts number.<br /><br />3. It seems the code also suffers from 'lemming effect' and probably pause instruction before transaction restart can make the program behave better. Probably you find the recent article from Andi Kleen, https://software.intel.com/en-us/articles/tsx-anti-patterns-in-lock-elision-code, useful.<br /><br />Lastly TSX is very self-willed technology. It absolutely doesn't like context switches and large/long transaction code. Also errata was found recently in the technology (http://www.anandtech.com/show/8376/intel-disables-tsx-instructions-erratum-found-in-haswell-haswelleep-broadwelly).Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-48578704433742451942014-10-13T08:48:01.507-06:002014-10-13T08:48:01.507-06:00Hey Alexander!
I try to use RTM instructions, usi...Hey Alexander!<br /><br />I try to use RTM instructions, using pthreads in a c file, but it always aborts<br />before entering the transaction. <br />Have you any idea why this is happening?<br />Thank you a lot in advance!<br /><br /><br />void *addone(void *arg)<br />{<br /> int i;<br /> unsigned status_tsx ;<br /><br /> while(1) { <br /><br /> status_tsx = _xbegin() ;<br /> <br /> <br /> if (status_tsx == _XBEGIN_STARTED ){<br /><br /> if(pthread_mutex_trylock (&mutexsum) !=0) {<br /><br /> _xabort(_ABORT_LOCK_BUSY);<br /><br /> }<br /><br /> else {<br /><br /> pthread_mutex_unlock (&mutexsum);<br /><br /> for (i=0;i<100;i++) sum = sum + 1.0;<br /><br /> <br /> pthread_exit(NULL);<br /> <br /> _xend();<br /> <br /> return; <br /> }<br /> <br /> }<br /><br /> if (!(status_tsx & _XABORT_RETRY)<br /> && !(status_tsx & _XABORT_CONFLICT)<br /> && !((status_tsx & _XABORT_EXPLICIT)<br /> && _XABORT_CODE(status_tsx) != _ABORT_LOCK_BUSY))<br /> break;<br /><br /> ++retries;<br /> <br /> }<br /><br /> pthread_mutex_lock (&mutexsum);<br /> for (i=0;i<100;i++) sum = sum + 1.0;<br /> pthread_mutex_unlock (&mutexsum);<br /> <br /> pthread_exit(NULL);<br /> <br />}Anonymoushttps://www.blogger.com/profile/02676886712877688526noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-56025251056101591412014-05-29T10:24:29.752-06:002014-05-29T10:24:29.752-06:00Hi Sahila,
sorry for late reply.
I'd expect ...Hi Sahila,<br /><br />sorry for late reply.<br /><br />I'd expect different results running on 4-core CPU - there is higher contention between the cores and/or hardware threads and we saw that TSX is very sensitive to number of concurrent threads...<br /><br />However, i7-4650U is the only Haswell CPU with which I made the experiments, so I'd be happy to see the benchmarks numbers for i7-4770 or other Haswell CPUs.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-3743539806726015032014-05-19T01:15:47.552-06:002014-05-19T01:15:47.552-06:00Hi
If I run the same code on Intel TSX i4770 for t...Hi<br />If I run the same code on Intel TSX i4770 for threads ranging from 2 to 8, will it give correct results ? Or is the code written above is specific to dual core processors only.<br />i4770 has 4 cores and can run 8 concurrent threads at a time.Anonymoushttps://www.blogger.com/profile/11286532715477338384noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-12216753125125684092014-02-14T01:09:57.143-07:002014-02-14T01:09:57.143-07:00You're welcome. I'm glad that the info was...You're welcome. I'm glad that the info was useful for you.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-23085382505374705772014-02-13T09:38:09.742-07:002014-02-13T09:38:09.742-07:00Alexander you where actually right. One can add th...Alexander you where actually right. One can add the instruction bellow if he is using mutex and not spin locks:<br /><br />if (pthread_mutex_t.__data.__lock != 0) _xabort ();<br /><br />Thank you for all your help and info!Anonymoushttps://www.blogger.com/profile/10913995668605720338noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-63600329037948714052014-01-17T06:11:40.571-07:002014-01-17T06:11:40.571-07:00Yes, your scenario is clear and I agree with it.
...Yes, your scenario is clear and I agree with it.<br /><br />Basically, a lock (spin lock in my case and mutex in yours) is just a memory area, which is updated when the lock is acquired, so we add it to transaction read set to be sure that nobody acquires the lock simultaneously with the transaction. Also we check that the lock isn't acquired because current thread can preempt a thread working under the lock or, as you pointed out, just run concurrently on other CPU.<br /><br />Could you rephrase "without forcing the mutex to lock in the case that it was free in the first place..."?<br /><br />I just had a quick look at glibc mutex implementation, so probably I'm wrong, but hope the following will be helpful. __pthread_mutex_lock() from glibc2.18/nptl/pthread_mutex_lock.c calls LLL_MUTEX_LOCK(mutex) or LLL_MUTEX_LOCK_ELISION(mutex) and the both operates with pthread_mutex_t.__data.__lock. Also the last one leads to __lll_lock_elision() and the __lock member (passed as futex) is checked in TSX transaction to know whether the lock is busy or not. So probably you should use pthread_mutex_t.__data.__lock in your transactional context to synchronize hardware transaction with mutex.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-73259780745733423902014-01-17T05:09:23.353-07:002014-01-17T05:09:23.353-07:00Thank you for your answer Alexander,
I totally ag...Thank you for your answer Alexander,<br /><br />I totally agree with you that a fallback path must exist. What I was questioning, was the need for synchronization between a thread using locks and a thread using RTM. <br /><br />My first though was that, if the thread using locks reads or writes the critical section then the thread that is using RTM will abort, because it will detect the conflict. However it seems to be that one situation where: <br />1)the thread using locks, will acquire the lock and read the read set.<br />2)then the thread using RTM will start and successfully commit(as no other thread is reading or writing the critical section at that time), changing the values in the write set<br />3)the thread with the locks will continue its calculations and overwrite the write set<br />(I hope I was clear with my example)<br /><br />So yes, it seems that there must be a synchronization between the two threads.<br /><br />As I said before, I am using mutexes but I couldn't find a way to add the mutex in the read set, without forcing the mutex to lock in the case that it was free in the first place... Any idea how I can do that?Anonymoushttps://www.blogger.com/profile/10913995668605720338noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-64040262197512585792014-01-15T12:34:25.873-07:002014-01-15T12:34:25.873-07:00Hi George,
let's consider two threads: one of...Hi George,<br /><br />let's consider two threads: one of them is currently running TSX transaction while the other one falls back to spin lock due to some reason (it happens fairly frequently with TSX). Thus we have to synchronize both the threads, i.e. spin lock in my case and mutex in your case must be synchronized with RTM which doesn't know anything about the lock and which operates only with memory locations. To do so we add the spin_lock to read set of the transaction - if it is locked or changed during transaction (after our check but before transaction commit) then transaction aborts.<br /><br />I started my TSX learning also from transactions without fallback at all. To make the program safe we need to rerun transactions until there is successful iteration. I found that this way has worse performance than fallback to spin-lock. Moreover, I've seen that the program goes into infinite loop (at least it was running too long), i.e. hardware transactions were aborting and aborting and aborting....Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-77324421697927674962014-01-15T10:08:44.048-07:002014-01-15T10:08:44.048-07:00In execute_short_trx function, a "hacky"...In execute_short_trx function, a "hacky" check is used to decide if the lock is used. Is it necessary to make this check and abort the RTM if the lock is used? And if yes why? Is there a similar way to check if a mutex (rather than a spinlcok) is locked<br />I am making the above question because I am testing a program of mine and i have the following problem. The program executes RTM and uses a mutex-lock fallback path if the RTM aborts more than 10 times. But the results i take show violation of coherence<br />I have the same program with no fallback path and only RTM and runs fine Anonymoushttps://www.blogger.com/profile/10913995668605720338noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-11848022691870847292014-01-10T07:01:03.583-07:002014-01-10T07:01:03.583-07:00Hi Sasha,
results for figure 3 are produced by fi...Hi Sasha,<br /><br />results for figure 3 are produced by first loop:<br /><br /> for (int trx_sz = 32; trx_sz <= 1024; trx_sz += 4)<br /> run_test(1, trx_sz, 1, 0, iter, Sync::TSX);<br /><br />so the upper limit for trx_sz is 1024, which gives us in trx_func()'s inner loop maximum transaction size as 2048 cache lines.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-14992045173391871032014-01-09T04:36:43.282-07:002014-01-09T04:36:43.282-07:00Regarding Figure 3.
If I understand your code cor...Regarding Figure 3. <br />If I understand your code correctly there is no real work for transactional memory beyond 256 CL( really 512 CL) so fluctuation around this point might be related to branch misspredictions and other burden of complex logic that works when abort probability is high but not 100%<br />Sashahttps://www.blogger.com/profile/12213591032791426046noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-32583615486319617412013-12-12T09:28:45.538-07:002013-12-12T09:28:45.538-07:00Slotty, thank you very much for the comment!
I...Slotty, thank you very much for the comment!<br /><br />I've conclusions in "Aborts on Single-threaded workload" according to your notice. So actually, TSX transactions are limited by L1d cache size.Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-27199821622393343252013-12-09T19:54:34.344-07:002013-12-09T19:54:34.344-07:00The article is really helpful. Regarding to questi...The article is really helpful. Regarding to question 1, I think as each transaction needs 2 cacheline (debit, credit), 32K cache can only support 256 transaction size at most in best situation. slottyhttps://www.blogger.com/profile/05198583977330839195noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-85400390780443965382013-11-18T05:49:12.328-07:002013-11-18T05:49:12.328-07:00Thank a lot, good job.Thank a lot, good job.Anonymoushttps://www.blogger.com/profile/07855567733551188197noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-39043834330705310082013-11-15T03:14:42.567-07:002013-11-15T03:14:42.567-07:00I didn't conclude the benchmarks since there a...I didn't conclude the benchmarks since there are 3 important questions:<br /><br />1. We see huge jump of transactional aborts when transaction work set reaches 256 cache lines (Figure 1). This is 16KB which is just only a quarter of L1d cache. The workload is single threaded, running strictly on one CPU core. Also I didn't use HyperThreading. I know that the cache has 8-way associativity, however I used static memory allocations (which are continous in virtual memory and likely continous physically) and it's unlikely that I have too many collisions in cache lines to get only 16KB of available cache memory for the transaction. So how the spike can be explained? Also Figure 3 (dependency of execution time on transaction size) shows strange fluctuation around 256 cache lines. Why execution time firstly quickly raises, then jumps down and after that smoothly increases again? There is no other such fluctuations on the graph.<br /><br />2. Test with overlapping data has shown very confusing results: TSX performance doesn't show peformance increasing for transactions with small data contention in comparison with transactions which modify the same data set on different CPUs. Probably I'm wrong with my fallback code and TSX transaction always finds the lock acquired. But I did special steps (retry transaction on _XABORT_RETRY, _XABORT_CONFLICT and _XABORT_EXPLICIT with my abort code events) to retry transaction more times instead of fallback to spin lock. In this scenario I'd expect smaller number of aborts for transactions with no data ovelapping in comparison with transactions with tight contention, but abort rates are the same. Why?<br /><br />3. And finally, I was experimenting with fallback condition a lot and results in the post are the best. How can I reduce number of transaction aborts (especially in conflict aborts for non-overlapping or with very small data overlapping transactions) and increase overall transactional concurrency?<br /><br /><br />I asked the question on Intel forum (http://software.intel.com/en-us/forums/topic/488911), but there is no answers. So research to be continued... :)Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-41104852591703882552013-11-15T03:02:02.854-07:002013-11-15T03:02:02.854-07:00Hi Sean,
nice to see you :)
Figures 4 and 6 depi...Hi Sean,<br /><br />nice to see you :)<br /><br />Figures 4 and 6 depict 2 thread (1 thread per CPU core) for spinlock and TSX. There is no data overlapping. I did exactly what you described - just updated two different (always different) memory locations (parts of huge array) in the threads. And TSX outperforms spinlock only on small transactions.<br /><br />There is fundamental problem - we need spinlock fallback. TSX can abort transaction due to many reasons, so we need to fallback to spinlock if a transaction aborted. If we just rerun transaction for all kinds of aborts, then we get very very smal performance. The system can progress so slow than you can think that it hangs... So falling back spinlock is that shared memory area on which we have contention event if we modify always different pieces of large array...Alexander Krizhanovskyhttps://www.blogger.com/profile/00939006050444455233noreply@blogger.comtag:blogger.com,1999:blog-2935119175410671378.post-19889005572596513672013-11-12T12:11:20.270-07:002013-11-12T12:11:20.270-07:00Not surprised by the performance.. transactional ...Not surprised by the performance.. transactional memory isn't really so much about performance, but more about making programming easier..<br /><br />One thing to think about, is the case when transactional memory typically does show better performance than locks.. that happens when there are many different shared locations, but two CPUs rarely try to update the same location at the same time.<br /> One way to test this case would be to create a very large array, then have each CPU randomly choose a cache-line segment from it, which it modifies. For spin lock, it will have to lock the entire array, or else divide the array into chunks and create a separate lock for each chunk. If it locks the entire array, then I would expect transactional memory to always win. But if there is a separate lock for each cache line sized chunk, then I would expect spin locks to win. It might be interesting to see at which size of block between those extremes the two cross over.<br /> The other thing is that the more CPUs there are, the better I would expect transactional memory to be. That's because locks serialize, and the more CPUs, the longer each has to wait.. it's multiplicative, so 4 CPUs means 3 wait to 1 compute, while 8 CPUs means 7 to 1.. but in contrast, transactional memory only depends on conflicts in accesses (in theory, at least.. I have no idea why all the conflicts were happening in the case of independent locations!).. so, it will only degrade if the greater amount of computation causes a proportionally larger number of conflicts (this depends on the nature of the application, and the nature of those unknown aborts)..<br /><br />Thanks for running these experiments and sharing!Seanhttps://www.blogger.com/profile/06077766802864255123noreply@blogger.com