Experiments with new low latency PyPy garbage collector in a thread.

TLDR; less performance spikes, faster average performance, PyPy is a good Python for games and multimedia apps on desktop systems, running GC in a thread is a good idea for GIL releasing apps.

In March 2018 at the PyPy winter sprint I spent some days working on improving pygame support on PyPy. Part of this was investigating pypy frame drop. In there I found PyPy had both lower average performance in this pygame-using-CPython-API benchmark, and had spikes which caused frame drops. Remember that in latency sensitive applications performance spikes are not allowed. Note that not very much of this code is pure python (where PyPy is fastest), so it's almost a worst case benchmark for PyPy performance.

Thanks to Antonio, the PyPy team and their supporters things have much improved since then. One improvement in particular is that PyPy now allows more control of when the Garbage Collection does its work. Read more about it in the blog post "PyPy for low latency systems".

In this post I'm going to compare my experiments with pygame with the experiments done last year. Additionally I'm going to try a new thing to improve performance which wasn't possible last year.

Running garbage collection in a thread.

Now that we can call garbage collection when we want, how about we do it when we release the GIL and do some other processing in C? PyPy doesn't run the garbage collector in a thread by default, but we can do that ourselves.

Remember that python has a GIL that can be released when you want to do IO or CPU intensive work. Here is an example program which has a thread waiting to be told when to run the incremental garbage collection step.

# Run pypy GC in background.
from Queue import Queue
from threading import Thread
import pygame

def gc_collect_step(q):
    """ this calls the gc in a thread.
    """
    while True:
        print '5) GC THREAD: Waiting to do gc.'
        ready_to_go = q.get()
        print '4) GC THREAD: calling gc_collect_step'
        if hasattr(gc, "collect_step"):
            gc.collect_step()
        q.task_done()

gc_queue = Queue()
worker = Thread(target=gc_collect_step, args=(gc_queue,))
worker.setDaemon(True)
worker.start()

pygame.display.set_mode((640, 480))
clock = pygame.time.Clock()
while True:
    print('1) at top of loop')
    pygame.event.get()
    gc_queue.put(True)

    print('2) tell GC thread to collect')
    print('3) flipping display. GIL released')
    pygame.display.flip()
    print ('6) finished flipping. GIL grabbed')
    clock.tick()
print(clock.get_fps())


This is how the program executes. Notice how it does display.flip and waits on the OS to do VSYNC at some refresh rate (usually 60FPS). In this spot many games and multimedia apps have some spare time to do other work.

Waiting for VSYNC is the perfect spot to do GC!

Trace of program execution when doing GC in background thread.
1) at top of loop
2) tell GC thread to collect
3) flipping display. GIL released
4) GC THREAD: calling gc_collect_step
5) GC THREAD: Waiting to do gc.
6) finished flipping. GIL grabbed
1) at top of loop
2) tell GC thread to collect
3) flipping display. GIL released
4) GC THREAD: calling gc_collect_step
5) GC THREAD: Waiting to do gc.
6) finished flipping. GIL grabbed
1) at top of loop
2) tell GC thread to collect
3) flipping display. GIL released
4) GC THREAD: calling gc_collect_step
5) GC THREAD: Waiting to do gc.
6) finished flipping. GIL grabbed
On my bench-marking setup this gives an increase of 11% frames per second on average. From 42 FPS to 47 FPS on average. Note, for this benchmark python2.7 gets 51 FPS on average.

Running garbage collection in a thread should be able to improve the performance on pypy of many apps which wait on the GIL.


Our performance goal.


Our performance goal is very little variance between frames and no spikes above 0.013. 0.013 is about the amount of time we have to do our work before we ask the frame to be flipped at 60 times per second.

Python 2.7 baseline measurement.


So we have something to compare against, here is a measurement of Python 2.7.

Note various spikes in performance, 51 FPS on average with many frames falling in the 0.005 to 0.007 range (below the required 0.013). The the funny spike at the start. I'm not sure why this is happening now compared to when I measured last year. The underlying OS has changed, or perhaps some other software is running on my computer.

Remember: Our performance goal is very little variance between frames and no spikes above 0.013. 0.013 is about the amount of time we have to do our work before we ask the frame to be flipped.

Python 2.7 isn't that bad here. It doesn't spike over 0.013 but it does have quite a lot of variance in some frames.


PyPy 6.0.0 (last release, does not include new GC stuff).

What about PyPy 6.0.0 which came out some months after my experiments last year. Here we see 0.005 to 0.010 with some spikes of around 0.0125 and higher. This older version of PyPy fails the performance goal.

PyPy 6.0.0 gc.disable().


What if we just disable the garbage collector entirely? (Apart from memory growing until the program starts swapping or crashes). After a few seconds it just starts to slow down a lot.


Time per frame on PyPy nightly (6.1.0 alpha0). gc.disable(), gc.collect_step().


This is using gc.disable() at the top of the main loop, and calling the new gc.collect_step() just before display.flip when most of the work has been done already. Remember that display.flip releases the GIL and waits for VSYNC at 60FPS.

Here we see the average FPS at around 42 with times mostly between 0.006 and 0.008. At the beginning and end there are some spikes. But mostly things stay below the magical 0.015 seconds required to get below 60 FPS. Here is another measurement.

With this second measurement we see that something is going on at the start and at the end again.  Again, mostly there are few spikes above 0.015 and things mostly stay between 0.006 and 0.008. This capture does show one spike, but I found this hard to reproduce.

This is a great improvement over CPython in all except average FPS. The timings in the graphs are the work that is done before display.flip is called.

PyPy nightly (6.1.0 alpha) GC in a thread, gc.disable().


Finally we have the GC running in a thread. This gives us 47.5 FPS on average. Most times stay between 0.6 and 0.8, however now there is a large number of spikes. None of them go above 0.015, but quite a few go above our goal of 0.013.

Why are these spikes here? I'm not sure. It needs to be investigated some more.
I ran the benchmark a few more times, and it seemed to spike less.




So then I switched to using threading.Event instead of threading.Queue, and there seemed to be less spikes over multiple runs. Probably using a lower level synchronization primitive (like threading.Lock) could improve things more. Also using a lower level primitive at C level would probably work even better. Additionally I'm not using the pypy36 which has some (hopefully) improved threading.Event and threading.Queue implementations.



Here's the example using Event instead of Queue.

from threading import Event, Thread, Lock
import pygame

def gc_collect_step(event):
    """ this calls the gc in a thread.
    """
    while True:
        print '5) GC THREAD: Waiting to do gc.'
        event.wait()
        print '4) GC THREAD: calling gc_collect_step'
        if hasattr(gc, "collect_step"):
            gc.collect_step()
        event.clear()

gc_event = Event()
worker = Thread(target=gc_collect_step, args=(gc_event,))
worker.setDaemon(True)
worker.start()


pygame.display.set_mode((640, 480))
clock = pygame.time.Clock()
while True:
    print('1) at top of loop')
    pygame.event.get()
    gc_event.set()

    print('2) tell GC thread to collect')
    print('3) flipping display. GIL released')
    pygame.display.flip()
    print ('6) finished flipping. GIL grabbed')
    clock.tick()
print(clock.get_fps())


Also, you can find the pypy-bench-testsprite.py in a gist. It needs to be run within the pygame examples folder.

PyPy meets performance goals for games

In conclusion:
/Happy dance/
The results that Antonio posted in "PyPy for low latency systems" also appear in the experiments I made. Worst case performance can be increased, and the performance spikes can be removed.

PyPy meets the performance goals for games now if you disable the gc at the top of the main loop, and call the gc collection every frame. The spikes that were there before are gone. The variance in performance is actually less than CPython.

Calling GC in a thread is a viable option which increases the average FPS very close to CPython levels. However some more work needs to be done to try and find a way that avoids the increased performance jitter. Note, that this jitter is still at acceptable levels.

Running the GC in a thread can probably be used for a peak performance increase by many apps  which release the GIL for IO or CPU intensive operations.

I'd like to remind people that this benchmark is a worst case for PyPy and games. In that there is also CPython API used rather than CFFI(fasted ffi on PyPy), and the benchmark contained not much pure python code. If you were also doing physics, collision detection and other stuff in pure python PyPy would pull ahead. Also this was using pygame with SDL1 so that I could sort-of compare performance to the previous time. Pygame 2 with SDL2 runs this benchmark much faster because of hardware acceleration. Another reminder that PyPy 6.1.0 isn't released yet, so the results will differ compared to the nightly version.

The combination of PyPy's JIT with the Cython ahead of time compiler makes a formidable combination. I'm looking forward to using them all with the hardware acceleration that pygame 2 provides (Vulkan, Metal, OpenGL, Direct3d, ...).

Comments

Nikita Soni said…
Xfinity Authorize and Xfinity.com/authorize is an official website that help users to activate Xfinity Gateway. xfinity.com/authorize User can get the best plans online for the Internet, video and mobile services too. Get the package of entertainment by connecting with it and also get any services online whatever you need. Xfinity locates offline financial deals for authorized Internet, video and mobile services. You can activate Xfinity Authorized by visiting xfinity.com/authorize the website Xfinity.com/authorize. And with xfinity you can get the service you need offline. You can visit our website xfinity.com/authorize for more information.
Nikita Soni said…

Attempt office.com/setup which is extremely simple to install, download and recover. Utilization of it is additionally straightforward and the client can become familiar with the office.com/setup utilization of it without any problem. Online Support&help alternative is likewise accessible in all application which gives a moment rule.Microsoft Office Setup is the office.com/setup complete bundle of Microsoft programs as it takes to the a variety of jobs, servers, and affiliations like PowerPoint, Excel, Word, Outlook, Publisher, OneNote, and Access.A progressing Microsoft statement gives some comprehension of how the association designs the office.com/setup landing of new Office 365 ProPlus features. Microsoft uses the equivalent "channels" language for MS Office 365 ProPlus incorporates revives as it does with Windows 10 feature invigorates.
Nikita Soni said…
McAfee inception with Mcafee antivirus gives all-around protection blocking all the bothersome perils, malware mcafee.com/activate disease which can hurt your PC.Introduce mcafee antivirus in your PC with high class experts and best tech group. Simply ring us and we are prepared to help you till the last moment of establishmentmcafee.com/activate visit here.
Nikita Soni said…
Avg.com/retail - Steps to Download, Install & Activate AVG Retail antivirus to safeguard your avg.com/retail computer and other devices from online Trojan risk, cyber threats.

Download norton antivirus to make your computer virus norton.com/setup free with the best support and tech team. Feel free to contact us
Nikita Soni said…
AVG key activation is a must through www.avg.com/retail webpage. avg.com/retail AVG antivirus offers advanced security program that blocks threats, identity theft viruses. avg.com/retail Know step by step process how to download, install and activate the AVG product. Enter your AVG license for AVG Retail. avg.com/retail If you need any assistance related to activation and installation of your AVG Antivirus you can visit at avg.com/retail.

Avg retail activation the technical world is improvising day by avg.com/retail day and as per the technology is improvising the online.A Complete Guide of avg.com/retail antivirus. AVG antivirus is the most trusted and prominent brand among other antivirus.AVG Antivirus, developed avg.com/retail AVG Technologies, is antivirus software that comes with highly advanced security tools. AVG is good for malware protection for AVG.
Nikita Soni said…
Follow the below-stated steps to activate Amazon Prime Video on your Smart TV now: GO to the home page of the Smart TV and search for Amazon Prime Video. Amazon App will open in front of you. Here you will amazon.com/mytv find “Register on the Amazon Website” and “Sign in and Start”.Amazon Video was previously called Amazon Video and before that amazon.com/mytv it was called Amazon Instant Video. Through Amazon Prime you can enjoy new and old movies and TV service. Amazon Prime is very well known for its name as well as its services. Amazon Prime is the best amazon.com/mytv American Internet video on demand service.It is also available on a few selected set-top boxes at a cheaper cost.GO to the home page of the Smart TV and search for Amazon Prime Video.

Prime Video was previously called Amazon Video and before that it was called Amazon Instant Video. Through Amazon Prime you can enjoy new and old movies and TV service. Amazon Prime is very primevideo.com/mytv well known for its name as well as its services. Amazon Prime is the best American Internet video on demand service.
Nikita Soni said…
Mcafee.com/activate- Find the easy way to download, install and activate McAfee Antivirus, type the 25-digit product key at mcafee.com/activate Protect your PC from virus, spam, spyware, and malware. McAfee provides the latest antivirus software, spyware removal and internet security for your home or home office.

Amazon.com/mytv is a video streaming service available for Amazon Prime members. With an eligible Amazon Prime membership, you have access to thousands of Prime Video titles at no additional cost.Choose Register option on the Amazon website- to get a 5–6 character code, then sign in to your Amazon account and enter your code to enjoy watching your favorite movies at. amazon.com/mytv If you are facing any kind of issue regarding amazon tv registration, login or any other troubleshooting visit to get the best and simple way to resolve all your issues. We provide 24*7 services to our users.
Nikita Soni said…
For account and technical support directly from McAfee's award winning mcafee.com/activate Service and Support Website. Get help via MVT, FAQs, and live support via chat and phones.

my.Roku.com is the free official site to link, activate, set-up and manage roku.com/link your Roku player or Roku TV. Roku never charges for linking or set-up support.Steps to activate Roku Device · Open web browser, go to the Roku.com/link · Enter the 4 digit code displayed on your TV /Device· roku.com/link Fill in all the details like name.
Nikita Soni said…

Amazon.com mytv is a registration portal of Amazon Prime amazon.com/mytv from which members can watch thousands of movies and TV shows.

Giving users an unparalleled streaming experience, Team roku.com/link takes great pride in being the number one streaming service providers in the world.

To Activating Hulu, you have to visit www.hulu.com/activate. Enter the code offered on your hulu.com/activate TV screen and clue in to your Hulu account. Article by Activate.
Nikita Soni said…

Norton web security is commonly used antivirus gives the least requesting to use and most intutive affirmation for your PC and your mobiles .present it and negligence viruses,spyware,root-units - , Download norton 360 hackers.for more nuances visit. norton.com/setup To enact the Norton setup, select the Activate Now option at the base. To recharge the membership norton.com/setup for Norton, select the Help choice and snap on Enter item key. Cautiously type the right Norton item key in the clear. Snap on the Next catch.Go through with for more details norton.com/setup Download Norton Mobile Security and Antivirus application that can shield your records from getting affected from any online malware or contamination norton setup product key.

Go to the Norton Security Online page and click Get Norton Security Online. Type in your Xfinity ID and password, if you're asked. Create a Norton account, then sign in. Choose whether you want to install [norton.com/setup it on this device or another one. Start your installation. Click Run. Let the program run. If Windows asks for permission, click Yes.Norton help to verify your devices against antiviruses and help to progress and keep up your information from compitetior. Present norton.com/setup norton in your pc click here for nuances.To install Norton Security Online on a different computer, log in to that computer and then download Norton. norton.com/setup Comcast now offers Norton Security Online instead of Norton Security Suite, and it is available at the Xfinity website. You are not required to update to Norton Security Online. Norton Security Online has a new look and feel.
Nikita Soni said…
Download Norton Security. Manage Users & Alerts. Reset My Password. Find My Account Number. Monitor Home Security. Upgrade My Service. Find My Xfinity ID. Get Help & Support. Program My Remote. norton.com/setup Submit Feedback. Find an Xfinity Store. Move My Services. Run Internet Speed Test. Bundles & Promotions. Customer Guarantee. Compare the Competition.Norton Internet Security gave you virus protection. Norton 360 gives you much more. Norton 360 plans give you device security to help protect PCs, Mac® and mobile devices against norton.com/setup viruses and malware, plus new ways to help protect your devices and online privacy – all in a single solution.If you want to download and install Norton on Windows, then you can follow the provided steps: Open the internet browser and input norton.com/setup. Click on Sign In to the account. norton.com/setup Go to the Norton download link on the Get Started page.
Nikita Soni said…

The HBO Max content offering is a superset of what was available hbomax.com/tvsignin under the Now moniker, with the same $14.99 monthly subscription price.HBO Max is a new streaming service with an extensive library. hbomax.com/tvsignin Here's an HBO Max show and movies list including original series and more.To complete the HBO Max TV sign in process,hbomax.com/tvsignin choose the button below and then enter the code from your TV. The HBO Store offers free products sometimes ...

Complete the HBO Max TV sign in process by entering the code from your device. hbomax.com/tvsignin Watch and chat now with millions of other fans from around the world.‎HBO Max is the streaming platform that bundles all of HBO together with even hbomax.com/tvsignin more of your favorite TV series, blockbuster movies, plus new Max Originals.Complete the HBO Max TV sign in process by entering the code from your device. hbomax.com/tvsignin Catch hit movies, popular shows, live news, sports & more the web or on your ..
Nikita Soni said…
“Roku devices don't appear to stream when using AirPlay from the HBO Max app. hbomax.com/tvsignin I followed the directions and clicked the link that said hbomax.com/tvsignin WarnerMedia's new streaming service, HBO Max, is set to ... To complete the HBO Max TV sign in process, choose the button below. To complete the HBO Max TV sign in process, choose the button below and then enter the code from your TV. hbomax.com/tvsignin Signing up for an HBO subscription is not allowed ...

HBO Max is the latest streaming service to take on Netflix and Disney+. It includes hbomax.com/tvsignin content from HBO Go and HBO Now, WarnerMedia. HBO Max is a new subscription video streaming service launching on Samsung Smart TV on May 27th, which brings the best of HBO with more original hbomax.com/tvsignin content, past seasons of your favorite shows, and movies.Visit HBOMax.com or download the app on your hbomax.com/tvsignin Apple or Android device and login with your Optimum or Altice One account. 
Derek Lafortune said…
Do you need immediate assistance on how to reset your Account Live password? We are here to help you with the most viable solution that can help you for Account Live password reset. However, if you want to execute this process on your own, then account.live.com/password/reset is the place you Once this link opens, you have to just follow the onscreen instructions that will help you to recover Account Live. To begin, the first thing you have to do is to validate or verify yourself by answering the need to visit.
security questions correctly. Then you will receive a password reset link on your provided mobile number or email ID. Do exactly as instructed, and the way to Account Live password reset will become smoother.
Call our Windows 10 Support Number: 1-877-814-4455
Check Full Steps here- Change Windows 10 password
Derek Lafortune said…
Get your brand established in the internet market by optimizing it with the best SEO techniques. Choose Sarahbits, the Best SEO Company, and take your e-business to the new heights of success. We have the most qualified SEO professionals who are conversant with all the tactics and techniques required to make your website rank high in the top search engines.
Derek Lafortune said…
Dragon NaturallySpeaking software is a speech recognition program that allows the user to speak into a microphone on a computer with the software translating
dragon naturally speaking
Derek Lafortune said…
Are you looking for it daily, Monthly, weekly home cleaning services Columbus Ohio? So you are in the right place. We are providing the home cleaning and maid services near you at an affordable and efficient price list. You Book Online as well on call after the checking all price list because we have multiple price lists according to customer budgets and home length. 100% Clients Satisfaction Service and 24/7 customer service available. For more details call us: +1614 3522 809
house cleaning service near me
Derek Lafortune said…
I thank the author for these experiments. Their results will be useful to many programmers who use Python as their main language and are hesitant to switch to this language interpreter. PyPy is a compatible Python interpreter that is a decent alternative to CPython 2.7, 3.6, and soon 3.7. By installing and running your application with it, you can achieve a noticeable increase in speed. Keep in mind that PyPy's performance is directly dependent on what your code is doing. PyPy is slower in some situations. However, on average, it is 4.3 times faster than Python. More information about the interpreter itself and its application can be found in Facebook posts in which the authors tell you how to use it. I found there a couple of dozen posts on this topic and noticed that most often they had at least 30 thousand likes! I'm sure this is because their authors used the services https://soclikes.com/buy-facebook-likes to quickly increase the number of likes.
We have the most experienced and professional embroidery digitizers among all other embroidery digitizing companies in the USA. Our team of experts is highly experienced in the respective field of digitizing. Our designers are talented and experienced in the relevant area of the work. We make sure to follow the right procedures of digitizing embroidery while meeting the standards of custom embroidery designing.
joyalex said…
if you are facing issue in the qb company file due to firewall error so it can be resolved by the quickbooks update error by the file doctor and other 6 methods so it shows the during the errorquickbooks firewall error then it can be solved by the quickbooks update error

Mike Johnson said…
I think you can make a video about your experiment and post it on instagram. From here https://socialapples.com/instagram-will-hide-its-likes-completely-whos-benefiting-from-this-decision/ you will learn how to get some likes for your video
Family Leather is a well-known platform to produce top-notch quality large adult backpack made with the best quality material. They have a diverse variety of leather bags of all types. They have a wide variety of trendy collections of backpacks in Canada. To get the best quality leather bags for women, simply visit the official website
Qwik Aid said…
I appreciated this post, I will share it with my favorite people.
You are also a Verizon user. If you have forgotten your Verizon email password or need help to Recover Verizon Email Password, Qwik Aid Team is here to help. Be that because it may, if you face any problem, you'll contact Qwik Aid experts to assist you together with your Technical inquiry.
Unknown said…
I would like to bring this to the notice of the public about how I came in contact with Mr Pedro after I lost my job and was denied a loan by my bank and other financial institutions due to my credit score. I could not pay my children's fees. I was behind on bills, about to be thrown out of the house due to my inability to pay my rent, It was during this period my kids were taken from me by foster care. Then I set out to seek funds online where I lost $3,670 that I borrowed from friends which I was ripped off by two online loan companies. Until i read about Mr Pedro helping people with an loan online in which this email was stated (pedroloanss@gmail.com) somewhere on the internet, Still wasn't convinced because of what i have been through until a relative of mine who is a clergy also told me about the ongoing loan scheme  at a very low interest rate of 2% and lovely repayment terms without penalty for default of payment. I had no choice than to also contact them, which I did. Mr Pedro responded back to me. That day was the best and greatest day of my life which can never be forgotten when I received a credit alert for the $400,000.00 Usd loan amount I applied for. I utilized the loan effectively to pay up my debts and to start up a business and today my kids and I are so happy and fulfilled. You can also contact them through email: (pedroloanss@gmail.com / WhatsApp Text : +18632310632)   Why am I doing this? I am doing this to save as many that are in need of a loan not to be victims of scams on the internet. Thanks and God bless you all, I'm Oleksander Artem from Horizon Park BC , Ukrain.
cally jesper said…
Informative Post. Such a one of useful information. Thanks for sharing with us.
With skilled and trained electricians, Todd Peters Electric as a Electrician Corona Ca service provider has come to your doorstep with extreme electrical solutions. We provide quality Electrical Services with careful analysis and observation of client's demands and budgets. If you would like to hire an electrician, simply dial our toll-free number or visit our website.
solar panel said…
If you are looking for orange county solar companies at affordable or reasonable prices. Look no further, Burge Solar Power is one of the best solar panel company in California that specialises in installing the best solar panels at residential and commercial sites. For more information, you can call us at (951) 787-9800.
Karthikbk said…
Aimore Technologies is the best msbi certification training in Chennai with 6+ years of experience. We are offering online and classroom training. Visit Us:MSBI Certification Training in Chennai
Kane416 said…

The blog was really Excellent Keep up the good work.....
Event management services in chennai
How do I recover my cash app account without a password?
How do I recover my cash app account without a password? If you want to recover your cash app account after losing your password then this is the best place for you. Here you will know how to recover a cash app account without a password you have to do nothing to know this except a click on this link.
Slot said…
เต้
ambbet
Way cool! Some very valid points! I appreciate you writing this
write-up and also the rest of the site is very good.

토토사이트
토토
I couldn't resiust commenting. Perfectly written!

토토
토토사이트
스포츠중계
토토사이트

I might want to thank you for the efforts you have made in composing this post.

Zonahobisaya said…
Bra nettside : Zonahobisaya
Bra nettside : One Piece
Bra nettside : Zonahobisaya
Bra nettside : Zonahobisaya
Bra nettside : Resep Masakan
Bra nettside : Terbanyak
Bra nettside : lambang
Bra nettside : Zonahobisaya
Nathalie said…
I like your blog. Thanks for this. Keep on posting. https://licflooring.com/
Iyaz Khan said…
Suppose you're interested in using PyPy and the LLGC in a threaded application. In that case, it may be worthwhile to experiment with different configurations and measure performance to see if the benefits outweigh any potential drawbacks. Dissertation Writing Service
wow slot 888 โบนัสแตกดีแต่ละวัน ได้รางวัลใหญ่ไม่จำกัด ฝาก-ถอนไวที่สุดเพียงแค่ 5 วินาทีเพียงแค่นั้นเองลงทะเบียนวันนี้รับเครดิตฟรี 100 บาท pg slot เว็บที่มีเกมให้เล่นมากมาย
slot-king-777 สำหรับเพื่อนฝูงๆที่อยากเล่น เว็บไซต์ สล็อต ฝากถอนไม่มีขั้นต่ํา หรือ สล็อต เว็บไซต์ตรงไม่ผ่านเอเย่นต์ ไม่มีอย่างต่ำ พวกเรามีเกม PGSLOTออนไลน์

Popular posts from this blog

Draft 3 of, ^Let's write a unit test!^

Is PostgreSQL good enough?

post modern C tooling - draft 6