post modern C tooling - draft 6

Contemporary C tooling for making higher quality C, faster or more safely.

DRAFT 0 - 10/11/18, 
DRAFT 1 - 9/16/19, 7:19 PM, I'm still working on this, but it's already useful and I'd like some feedback - so I decided to share it early.
DRAFT 2 - 10/1/19, mostly additions to static analysis tools.
DRAFT 3 - 10/4/19, updates on build systems, package management, and complexity analysis. 
DRAFT 4 - 10/6/19, run time dynamic verification and instrumentation, sanitizers (asan/ubsan/etc), performance tools, static analyzers.
DRAFT 5 - C interpreter(s). 
DRAFT 6 - 11/6/19, mention TermDebug vim,  more windows debugging tools, C drawing for intro.



In 2001 or so people started using the phrase "Modern C++". So now that it's 2019, I guess we're in the post modern era? Anyway, this isn't a post about C++ code, but some of this information applies there too.


No logo, but it's used everywhere.


Welcome to the post modern era.

Some of the C++ people have pulled off one of the cleverest and sneakiest tricks ever. They required 'modern' C99 and C11 features in 'recent' C++ standards. Microsoft has famously still clung onto some 80s version of C with their compiler for the longest time. So it's been a decade of hacks for people writing portable code in C. For a while I thought we'd be stuck in the 80s with C89 forever. However, now that some C99 and C11 features are more widely available in the Microsoft compiler, we can use these features in highly portable code (but forget about C17/C18 ISO/IEC 9899:2018/C2X stuff!!). Check out the "New" Features in C talk, and the Modern C book for more details.

So, we have some pretty new language features in C with C11.  But what about tooling?

Tools and protection for our feet.

C, whilst a work horse being used in everything from toasters, trains, phones, web browsers, ... (everything basically) - is also an excellent tool for shooting yourself in the foot.

Noun

footgun (plural footguns)
  1. (informal, humorous, derogatory) Any feature whose addition to a product results in the user shooting themselves in the foot. C.

Tools like linters, test coverage checkers, static analyzers, memory checkers, documentation generators, thread checkers, continuous integration, nice error messages, ... and such help protect our feet.

How do we do continuous delivery with a language that lets us do the most low level footgunie things ever? On a dozen CPU architectures, 32 bit, 64bit, little endian, big endian, 64 bit with 32bit pointers (wat?!?), with multiple compilers, on a dozen different OS, with dozens of different versions of your dependencies?

Surely there won't be enough time to do releases, and have time left to eat my vegan shaved ice desert after lunch?



Debuggers

Give me 15 minutes, and I'll change your mind about GDB. --
https://www.youtube.com/watch?v=PorfLSr3DDI
Firstly, did you know gdb had a curses based 'GUI' which works in a terminal? It's a quite a bit easier to use than the command line text interface. It's called TUI. It's built in, and uses emacs key bindings.

But what if you are used to VIM key bindings? cgdb to the rescue.

https://cgdb.github.io/

VIM has integrated gdb debugging with TermDebug since version 8.1.

Also, there's a fairly easy to use web based front end for GDB called gdbgui
 (https://www.gdbgui.com/). For those who don't use an IDE with debugging support built in (such as Visual studio by Microsoft or XCode by Apple).





Reverse debugger

Normally a program runs forwards. But what about when you are debugging and you want to run the program backwards?

Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.

How do you tame non determinism to allow a program to run the same way it did when it crashed? In C and with threads some times it's really hard to reproduce problems.

rr helps with this. It's actual magic.

https://rr-project.org/






LLDB - the LLVM debugger.

Apart from the ever improving gdb, there is a new debugger from the LLVM people - lldb ( https://lldb.llvm.org/ ).


IDE debugging

Visual Studio by Microsoft, and XCode by Apple are the two heavy weights here.

The free Visual Studio Code also supports debugging with GDB. https://code.visualstudio.com/docs/languages/cpp

Sublime is another popular editor, and there is good GDB integration for it too in the SublimeGDB package (https://packagecontrol.io/packages/SublimeGDB).

Windows debugging

Suppose you want to do post mortem debugging? With procdump and WinDbg you can.

Launch a process and then monitor it for exceptions:
C:\>procdump -e 1 -f "" -x c:\dumps consume.exe
This makes some process dumps when it crashes, which you can then open with WinDbg(https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview).


Portable building, and package management

C doesn't have a package manager... or does it?

Ever since Debian dpkg, Redhat rpm, and Perl started doing package management in the early 90s people world wide have been able to share pieces of software more easily. Following those systems, many other systems like Ruby gems, JavaScript npm, and Pythons cheese shop came into being. Allowing many to share code easily.

But what about C? How can we define dependencies on different 'packages' or libraries and have them compile on different platforms?

How do we build with Microsofts compiler, with gcc, with clang, or Intels C compiler? How do we build on Mac, on Windows, on Ubuntu, on Arch linux? Sometimes we want to use an Integrated Development Environment (IDE) because they provide lots of nice tools. But maybe also three IDEs (XCode, Microsoft Visual C, CLion, ...) depending on platform. But we don't want to have to keep several IDE project files up to date. But we also want to integrate nicely with different OS packagers like Debian, FreeBSD. We want people to be able to use apt-get install for their dependencies if they want. We also want to cross compile code on our beefy workstations to work on microcontrollers or slower low memory systems (like earlier RaspberryPi systems).



The Meson Build System.

If CMake is modern, then The Meson Build System (https://mesonbuild.com/index.html) is post modern.
"Meson is an open source build system meant to be both extremely fast, and, even more importantly, as user friendly as possible.
The main design point of Meson is that every moment a developer spends writing or debugging build definitions is a second wasted. So is every second spent waiting for the build system to actually start compiling code."
It's first major user was GStreamer, a multi platform multimedia toolkit which is highly portable. Now it is especially popular in the FreeDesktop world with projects like gstreamer, GTK, and systemd amongst many others using it.

The documentation is excellent, and it's very fast compared to autotools or cmake.
https://www.youtube.com/watch?v=gHdTzdXkhRY


Example meson.build for example project polysnake (https://github.com/jpakkane/polysnake):
"A Python extension module that uses C, C++, Fortran and Rust"?
 
project('polysnake', 'c', 'cpp', 'rust', 'fortran',
  default_options : ['cpp_std=c++14'],
  license : 'GPL3+')

py3_mod = import('python3')
py3_dep = dependency('python3')

# Rust integration is not perfect.
rustlib = static_library('func', 'func.rs')

py3_mod.extension_module('polysnake',
  'polysnake.c',
  'func.cpp',
  'ffunc.f90',
  link_with : rustlib,
  dependencies : py3_dep)

IDEs are supported by exporting to XCode+Visual Studio, and they provide their own interface (which a few less well known IDEs are starting to use).


Conan package manager

There are several packaging tools for C these days, but one of the top contenders is Conan (https://conan.io/).
"Conan, the C / C++ Package Manager for Developers  The open source, decentralized and multi-platform package manager to create and share all your native binaries."
What does a CMake conan project look like? (https://github.com/conan-io/hello)
What does a conan Meson project look like? (https://docs.conan.io/en/latest/reference/build_helpers/meson.html)


CMake

"Modern CMake" is the build tool of choice for many C projects.

Just don't read the official docs, or the official book - they're quite out of date.
An Introduction to Modern CMake (https://cliutils.gitlab.io/modern-cmake/)
CGold: The Hitchhiker’s Guide to the CMake (https://cgold.readthedocs.io/en/latest/)

"CMake is a meta build system. It can generate real native build tool files from abstracted text configuration. Usually such code lives in CMakeLists.txt files."

Around 2015-2016 a bunch of IDEs got support for CMake: Microsoft Visual Studio, CLion, QtCreator, KDevelop, and Android Studio (NDK). And a lot of people tried extra hard to like it, and a lot of C projects started supporting it.

Apart from wide IDE support, it is also supported quite well by package managers like VCPkg and Conan.


Interpreter and REPL

Usually C is compiled.
Bic is an interpreter for C (https://github.com/hexagonal-sun/bic).

bic: A C interpreter and API explorer

Additionally there is "Cling" which is based on the LLVM infrastructure and can even do C++.
https://github.com/root-project/cling




Testing coverage.

Tests let us know that some certain function is running ok. Which code do we still need to test?

gcov, a tool you can use in conjunction with GCC to test code coverage in your programs.
lcov, LCOV is a graphical front-end for GCC's coverage testing tool gcov.


Instructions from codecov.io on how to use it with C, and clang or gcc. (codecov.io is free for public open source repos).
https://github.com/codecov/example-c


Here's documentation for how CPython gets coverage results for C.
 https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov

Here is the CPython Travis CI configuration they use.
https://github.com/python/cpython/blob/master/.travis.yml#L69
    - os: linux
      language: c
      compiler: gcc
      env: OPTIONAL=true
      addons:
        apt:
          packages:
            - lcov
            - xvfb
      before_script:
        - ./configure
        - make coverage -s -j4
        # Need a venv that can parse covered code.
        - ./python -m venv venv
        - ./venv/bin/python -m pip install -U coverage
        - ./venv/bin/python -m test.pythoninfo
      script:
        # Skip tests that re-run the entire test suite.
        - xvfb-run ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures
      after_script:  # Probably should be after_success once test suite updated to run under coverage.py.
        # Make the `coverage` command available to Codecov w/ a version of Python that can parse all source files.
        - source ./venv/bin/activate
        - make coverage-lcov
        - bash > (curl -s https://codecov.io/bash)




Static analysis

"Static analysis has not been helpful in finding bugs in SQLite. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis." -- https://www.sqlite.org/testing.html

According to David Wheeler in "How to Prevent the next Heartbleed" (https://dwheeler.com/essays/heartbleed.html#static-not-found the security problem with a logo, a website, and a marketing team) only one static analysis tool found the Heartbleed vulnerability before it was known. This tool is called CQual++. One reason for projects not using these tools is that they have been (and some still are) hard to use. The LLVM project only started using the clang static analysis tool on it's own projects recently for example. However, since Heartbleed in 2014 tools have improved in both usability and their ability to detect issues.

I think it's generally accepted that static analysis tools are incomplete, in that each tool does not guarantee detecting every problem or even always detecting the same issues all the time. Using multiple tools can therefore be said to find multiple different types of problems.


Compilers are kind of smart

The most basic of static analysis tools are compilers themselves. Over the years they have been getting more and more tools which used to only be available in dedicated Static Analyzers and Lint tools.
Variable shadowing and format-string mismatches can be detected reliably and quickly is because both gcc and clang do this detection as part of their regular compile. --  Bruce Dawson
Here we see two issues (which used to be) very common in C being detected by the two most popular C compilers themselves.

Compiling code with gcc "-Wall -Wextra -pedantic" options catches quite a number of potential or actual problems (https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html). Other compilers check different things as well. So using multiple compilers with their warnings can find plenty of different types of issues for you.

Compiler warnings should be turned in errors on CI.

By getting your errors down to zero on Continuous Integration there is less chance of new warnings being introduced that are missed in code review. There are problems with distributing your code with warnings turned into errors, so that should not be done.

Some points for people implementing this:
  • -Werror can be used to turn warnings into errors
  • -Wno-error=unknown-pragmas
  • should run only in CI, and not in the build by default. See werror-is-not-your-friend (https://embeddedartistry.com/blog/2017/5/3/-werror-is-not-your-friend).
  • Use most recent gcc, and most recent clang (change two travis linux builders to do this).
  • first have to fix all the warnings (and hopefully not break something in the process).
  • consider adding extra warnings to gcc: "-Wall -Wextra -Wpedantic" See C tooling
  • Also the Microsoft compiler MSVC on Appveyor can be configured to treat warnings as errors. The /WX argument option treats all warnings as errors. See MSVC warning levels
  • For MSVC on Appveyor, /wdnnnn Suppresses the compiler warning that is specified by nnnn. For example, /wd4326 suppresses compiler warning C4326.
If you run your code on different CPU architectures, these compilers can find even more issues. For example 32bit/64bit Big Endian, and Little Endian.


Static analysis tool overview.

Static analysis can be much slower than the analysis usually provided by compilation with gcc and clang. It trades off more CPU time for (hopefully) better results.


cppcheck focuses of low false positives and can find many actual problems.
Coverity, a commercial static analyzer, free for open source developers
CppDepend, a commercial static analyzer based on Clang
codechecker, https://github.com/Ericsson/codechecker
cpplint, Cpplint is a command-line tool to check C/C++ files for style issues following Google's C++ style guide.
Awesome static analysis, a page full of static analysis tools for C/C++. https://github.com/mre/awesome-static-analysis#cc
PVS-Studio, a commercial static analyzer, free for open source developers.


The Clang Static Analyzer

The Clang Static Analyzer (http://clang-analyzer.llvm.org/) is a free to use static analyzer that is quite high quality.
The Clang Static Analyzer is a source code analysis tool that finds bugs in C, C++, and Objective-C programs. Currently it can be run either as a standalone tool or within Apple Xcode. The standalone tool is invoked from the command line, and is intended to be run in tandem with a build of a codebase.
The talk "Clang Static Analysis" (https://www.youtube.com/watch?v=UcxF6CVueDM) talks about an LLVM tool called codechecker (https://github.com/Ericsson/codechecker).

On MacOS an up to date scan-build and scan-view is included with the brew install llvm.

$SCANBUILD=`ls /usr/local/Cellar/llvm/*/bin/scan-build`
$SCANBUILD -V python3 setup.py build

On Ubuntu you can install scan-view with apt-get install clang-tools.


cppcheck 

Cppcheck is an analysis tool for C/C++ code. It provides unique code analysis to detect bugs and focuses on detecting undefined behaviour and dangerous coding constructs. The goal is to detect only real errors in the code (i.e. have very few false positives).

The quote below was particularly interesting to me because it echos the sentiments of other developers, that testing will find more bugs. But here is one of the static analysis tools saying so as well.
"You will find more bugs in your software by testing your software carefully, than by using Cppcheck."
To Install cppcheck:
http://cppcheck.sourceforge.net/ and https://github.com/danmar/cppcheck
The manual can be found here: http://cppcheck.net/manual.pdf

brew install cppcheck bear
sudo apt-get install cppcheck bear

To run cppcheck on C code:
You can use bear (the build ear) tool to record a compilation database (compile_commands.json). cppcheck can then know what c files and header files you are using.

# call your build tool, like `bear make` to record. 
# See cppcheck manual for other C environments including Visual Studio.
bear python setup.py build
cppcheck --quiet --language=c --enable=all -D__x86_64__ -D__LP64__ --project=compile_commands.json

 It does seem to find some errors, and style improvements that other tools do not suggest. Note that you can control the level of issues found to errors, to portability and style issues plus more. See cppcheck --help and the manual for more details about --enable options.

For example these ones from the pygame code base:
[src_c/math.c:1134]: (style) The function 'vector_getw' is never used.
[src_c/base.c:1309]: (error) Pointer addition with NULL pointer.
[src_c/scrap_qnx.c:109]: (portability) Assigning a pointer to an integer is not portable.
[src_c/surface.c:832] -> [src_c/surface.c:819]: (warning) Either the condition '!surf' is redundant or there is possible null pointer dereference: surf.

/Analyze in Microsoft Visual Studio

Visual studio by Microsoft can do static code analysis too. ( https://docs.microsoft.com/en-us/visualstudio/code-quality/code-analysis-for-c-cpp-overview?view=vs-2017)

"Using SAL annotations to reduce code defects." (https://docs.microsoft.com/en-us/visualstudio/code-quality/using-sal-annotations-to-reduce-c-cpp-code-defects?view=vs-2019)
"In GNU C and C++, you can use function attributes to specify certain function properties that may help the compiler optimize calls or check code more carefully for correctness."
https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html

Custom static analysis for API usage

Probably one of the most useful parts of static analysis is being able to write your own checks. This allows you to do checks specific to your code base in which general checks will not work. One example of this is the gcc cpychecker (https://gcc-python-plugin.readthedocs.io/en/latest/cpychecker.html). With this, gcc can find API usage issues within CPython extensions written in C. Including reference counting bugs, and NULL pointer de-references, and other types of issues. You can write custom checkers with LLVM as well in the "Checker Developer Manual" (https://clang-analyzer.llvm.org/checker_dev_manual.html)

There is a list of GCC plugins (https://gcc.gnu.org/wiki/plugins) among them are some Linux security plugins by grsecurity.


Runtime checks and Dynamic Verification

Dynamic verification tools examine code whilst it is running. By running your code under these dynamic verification tools you can help detect bugs. Either by testing manually, or by running your automated tests under the watchful eye of these tools. Runtime dynamic verification tools can detect certain errors that static analysis tools can't.

Some of these tools are quite easy to add to a build in Continuous Integration(CI). So you can run your automated tests with some extra dynamic runtime verification enabled.

Take a look at how easy they are to use?
./configure CFLAGS="-fsanitize=address,undefined -g" LDFLAGS="-fsanitize=address,undefined"
make
make check

Address Sanitizer

Doing a test run with an address sanitizer apparently helps to detect various types of bugs.
AddressSanitizer is a fast memory error detector. It consists of a compiler instrumentation module and a run-time library. The tool can detect the following types of bugs:
  • Out-of-bounds accesses to heap, stack and globals
  • Use-after-free
  • Use-after-return (runtime flag ASAN_OPTIONS=detect_stack_use_after_return=1)
  • Use-after-scope (clang flag -fsanitize-address-use-after-scope)
  • Double-free, invalid free
  • Memory leaks (experimental)
How to compile a python C extension with clang on MacOS:
LDFLAGS="-g -fsanitize=address" CFLAGS="-g -fsanitize=address -fno-omit-frame-pointer" python3 setup.py install



Undefined Behaviour Sanitizer

From https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior detector. UBSan modifies the program at compile-time to catch various kinds of undefined behavior during program execution, for example:
  • Using misaligned or null pointer
  • Signed integer overflow
  • Conversion to, from, or between floating-point types which would overflow the destination
You can use the Undefined Behaviour Sanitizer with clang and gcc. Here is the gcc documentation for Instrumentation Options and UBSAN (https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html).
 
From https://www.sqlite.org/testing.html
To help ensure that SQLite does not make use of undefined or implementation defined behavior, the test suites are rerun using instrumented builds that try to detect undefined behavior. For example, test suites are run using the "-ftrapv" option of GCC. And they are run again using the "-fsanitize=undefined" option on Clang. And again using the "/RTC1" option in MSVC
To compile a python C extension with a UBSAN with clang on Mac do:
LDFLAGS="-g -fsanitize=undefined" CFLAGS="-g -fsanitize=undefined -fno-omit-frame-pointer" python3 setup.py install

Microsoft Visual Studio Runtime Error Checks

The Microsoft Visual Studio compiler can use the Run Time Error Checks feature to find some issues. /RTC (Run-Time Error Checks) (https://docs.microsoft.com/en-us/cpp/build/reference/rtc-run-time-error-checks?view=vs-2019)

From How to: Use Native Run-Time Checks (https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019)
  • Stack pointer corruption.
  • Overruns of local arrays.
  • Stack corruption.
  • Dependencies on uninitialized local variables.
  • Loss of data on an assignment to a shorter variable.

App Verifier

"Any Windows developers that are listening to this: if you’re not using App Verifier, you are making a mistake." -- Bruce Dawson
Stangely App Verifier does not have very good online documentation. The best article available online about it is: Increased Reliability Through More Crashes (https://randomascii.wordpress.com/2011/12/07/increased-reliability-through-more-crashes/)
Application Verifier (AppVerif.exe) is a dynamic verification tool for user-mode applications. This tool monitors application actions while the application runs, subjects the application to a variety of stresses and tests, and generates a report about potential errors in application execution or design.
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/application-verifier
https://docs.microsoft.com/en-us/windows/win32/win7appqual/application-verifier
https://docs.microsoft.com/en-us/security-risk-detection/concepts/application-verifier

  • Buffer over runs
  • Use after free issues
  • Thread issues including using TLS properly.
  • Low resource handling
  • Race conditions
If you have a memory corruption bug, App Verifier might be able to help you find it. If you are using Windows APIs wrong, have some threading issues, want to make sure you app runs under harsh conditions -- App Verifier might help you find it.

Related to App Verifier is the PageHeap tool(https://support.microsoft.com/en-us/help/286470/how-to-use-pageheap-exe-in-windows-xp-windows-2000-and-windows-server) It helps you find memory heap corruptions on Windows.




Performance profiling and measurement

“The objective (not always attained) in creating high-performance software is to make the software able to carry out its appointed tasks so rapidly that it responds instantaneously, as far as the user is concerned.”  Michael Abrash. “Michael Abrash’s Graphics Programming Black Book.”
Reducing energy usage, and run time requirements of apps can often be a requirement or very necessary. For a mobile or embedded application it can mean the difference of being able to run the program at all. Performance can directly be related to user happiness but also to the financial performance of a piece of software.

But how to we measure the performance of a program, and how to we know what parts of a program need improvement? Tooling can help.

Valgrind

Valgrind has its own section here because it does lots of different things for us. It's a great tool, or set of tools for improving your programs. It used to be available only on linux, but is now also available on MacOS.

Apparently Valgrind would have caught the heartbleed issue if it was used with a fuzzer.

http://valgrind.org/docs/manual/quick-start.html

Apple Performance Tools

Apple provides many performance related development tools. Along with the gcc and llvm based tools, the main tool is called Instruments. Instruments (part of Xcode) allows you to record and analyse programs for lots of different aspects of performance - including graphics, memory activity, file system, energy and other program events. By being able to record and analyse different types of events together can make it convienient to find performance issues.


LLVM performance tools.

Many of the low level parts of the tools in XCode are made open source through the LLVM project. See "LLVM Machine Code Analyzer" ( https://llvm.org/docs/CommandGuide/llvm-mca.html) as one example. See the LLVM XRay instrumentation (https://llvm.org/docs/XRayExample.html).

There's also an interesting talk on XRay here "XRay in LLVM: Function Call Tracing and Analysis" (https://www.youtube.com/watch?v=jyL-__zOGcU) by Dean Michael Berris.


GNU/Linux tools



Microsoft performance tools.


Intel performance tools.

https://software.intel.com/en-us/vtune




Caching builds

https://ccache.samba.org/

ccache is very useful for reducing the compile time of large C projects. Especially when you are doing a 'rebuild from scratch'. This is because ccache can cache the compilation of parts in this situation when the files do not change.
http://itscompiling.eu/2017/02/19/speed-cpp-compilation-compiler-cache/

This is also useful for speeding up CI builds, and especially when large parts of the code base rarely change.


Distributed building.


distcc https://github.com/distcc/distcc
icecream https://github.com/icecc/icecream


Complexity of code.

"Complex is better than complicated. It's OK to build very complex software, but you don't have to build it in a complicated way. Lizard is a free open source tool that analyse the complexity of your source code right away supporting many programming languages, without any extra setup. It also does code clone / copy-paste detection."
Lizard is a modern complexity command line tool,
that also has a website UI: http://www.lizard.ws/
https://github.com/terryyin/lizard

# install lizard
python3 -m pip install lizard
# show warnings only and include/exclude some files.
lizard src_c/ -x"src_c/_sdl2*" -w 

# Can also run it as a python module.
python3 -m lizard src_c/ -x"src_c/_sdl2*" -w

# Show a full report, not just warnings (-w).
lizard src_c/ -x"src_c/_sdl2*" -x"src_c/_*" -x"src_c/SDL_gfx*" -x"src_c/pypm.c"

Want people to understand your code? Want Static Analyzers to understand your code better? Want to be able to test your code more easily? Want your code to run faster because of less branches? Then you may want to find complicated code and refactor it.

Lizard can also make a pretty word cloud from your source.

Lizard complexity analysis can be run in Continuous Integration (CI). You can also give it lists of functions to ignore and skip if you can't refactor some function right away. Perhaps you want to stop new complex functions from entering your codebase? To stop new complex functions via CI make a supression list of all the current warnings and then make your CI use that and fail if there are new warnings.



Testing your code on different OS/architectures.

Sometimes you need to be able to fix an issue on an OS or architecture that you don't have access to. Luckily these days there are many tools available to quickly use a different system through emulation, or container technology.


Vagrant
Virtualbox
Docker
Launchpad, compile and run tests on many architectures.
Mini cloud (ppc machines for debugging)

If you pay Travis CI, they allow you to connect to the testing host with ssh when a test fails.


Code Formatting

clang-format

clang-format - rather than manually fix various formatting errors found with a linter, many projects are just using clang-format to format the code into some coding standard.



Services

LGTM is an 'automated code review tool' with github (and other code repos) support. https://lgtm.com/help/lgtm/about-automated-code-review

Coveralls provides a store for test coverage results with github (and other code repos) support. https://coveralls.io/




Coding standards for C

There are lots of coding standards for C, and there are tools to check them.


An older set of standards is the MISRA_C (https://en.wikipedia.org/wiki/MISRA_C) aims to facilitate code safety, security, and portability for embedded systems.

The Linux Kernel Coding standard (https://www.kernel.org/doc/html/v4.10/process/coding-style.html) is well known mainly because of the popularity of the Linux Kernel. But this is mainly concerned with readability.

A newer one is the CERT C coding standard (https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards), and it is a secure coding standard (not a safety one).

The website for the CERT C coding standard is quite amazing. It links to tools that can detect each of the problems automatically (when they can be). It is very well researched, and links each problem to other relevant standards, and gives issues priorities. A good video to watch on CERT C is "How Can I Enforce the SEI CERT C Coding Standard Using Static Analysis?" (https://www.youtube.com/watch?v=awY0iJOkrg4). They do releases of the website, which is edited as a wiki. At the time of writing the last release into book form was in 2016.







How are other projects tested?

We can learn a lot by how other C projects are going about their business today.
Also, thanks to CI testing tools defining things in code we can see how automated tests are run on services like Travis CI and Appveyor.

SQLite

"How SQLite Is Tested"

Curl

"Testing Curl"
https://github.com/curl/curl/blob/master/.travis.yml

Python

"How is CPython tested?"
https://github.com/python/cpython/blob/master/.travis.yml

OpenSSL

"How is OpenSSL tested?"

https://github.com/openssl/openssl/blob/master/.travis.yml
They use Coverity too: https://github.com/openssl/openssl/pull/9805
https://github.com/openssl/openssl/blob/master/fuzz/README.md

libsdl

"How is SDL tested?" [No response]

Linux

https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-testedhttps://www.linuxjournal.com/content/linux-kernel-testing-and-debugging

Haproxy

https://github.com/haproxy/haproxy/blob/master/.travis.yml



There is some discussion of Post Modern C Tooling on the "C_Programming" reddit forum.



pygame book

This article will be part of a book called "pygame 4000". A book dedicated to the joy of making software for making. Teaching collaboration, low level programming in C, high level programming in Python, GPU graphics programming with a shader language, design, music, tools, quality, and shipping(software).

It's a bit of a weird book. There's a little bit of swearing in it (consider yourself fucking warned), and all profits go towards pygame development (the library, the community, and the website).

Comments

Hamza said…
Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You clearly know what youre talking about, why waste your intelligence on just posting videos to your weblog when you could be giving us something enlightening to read?Arquitetos Residenciais São Paulo
Hamza said…
I have come across with so many sites but
dividata is one of it’s own kind!







Kaleem said…
Testosterone Lots of men choose to move toward the gym and start gaining muscle. So, for people to gain muscle Testosterone are recommended. Testogen


healer said…
I simply wanted to write down a fast word to mention you for those wonderful tips and hints you're showing on this site dg casino


I have shared this link with others to keep posting such information to provide the best in class homework help online at very affordable prices.
Our online assignment help Australia service is an online assignment help service provided by experienced Australian assignment help expert at here.
Swinburne University Of Technology Assignment Help
James Cook University Assignment Help
Murdoch University Assignment Help
Western Sydney University Assignment Help
University Of Canberra Homework Help
Charles Darwin University Assignment Help
Edith Cowan University Assignment Help
University Of New England Homework Help
Hamza said…
If you are looking for the best study platform you have landed on the perfect place!
essay writer service

Do follow this channel its useful its hard to find good ones. Thanks
a course in miracles



Hamza said…
Some of my friends have suggested me to visit such mind blowing sites,
This was the best experience!
Meet people in London
In such scenarios while getting stuck with any sort of technical or non-technical grievances in QuickBooks, simply call us on our QuickBooks Support Phone Number California +1(844)233-3033, and acquire exceptional accounting services from our executives. Our experts are skilled enough to answer all of the error codes users ask for.
QuickBooks Desktop Support +1(844)233-3033
QuickBooks Enterprise Support +1(844)233-3033
Quickbooks 24/7 Customer service +1(844)233-3033
Quickbooks Payroll Support +1(844)233-3033
QuickBooks Technical Support +1(844)233-3033
QuickBooks POS Support +1(844)233-3033
Quickbooks Support Phone Number USA
Quickbooks Support Phone Number USA
Helen Henson said…
Great one. I really like this post. Thanks for sharing.
Why you Need IT Services for your Small and Big Business
Hamza said…
One of the best sites i have visited in last few weeks and i would love to be the part of your emerging community!
먹튀검증업체
I think this is an informative post and it is very useful and knowledgeable. fertilizer and livestock trading and marketing





Hamza said…
This video amazon fba has something which will definitely expand your education and imagination,which will eventually make you more visionary!
Vinita Hotwani said…
McAfee inception with Mcafee antivirus gives all-around protection blocking all the bothersome perils, malware mcafee.com/activate disease which can hurt your PC.Introduce mcafee antivirus in your PC with high class experts and best tech group. Simply ring us and we are prepared to help you till the last moment of establishmentmcafee.com/activate visit here.
Vinita Hotwani said…
Go to hbomax.com/tvsignin - this redirects to https://activate.hbomax.com. hbomax.com/tvsignin Select your TV Provider. Sign in with the company User ID/email address. HBO Max is the hbomax.com/tvsignin premium streaming platform that bundles together all of HBO with even more of your favorite movies and TV shows from Warner Bros.HBO Max is the streaming platform that bundles all of HBO hbomax.com/tvsignin together with even more of your favorite TV series, blockbuster movies, plus new Max Originals.hbomax.com/tvsignin HBO Max is a premium streaming service that bundles all of HBO together with exclusive and licensed movies, television shows, and more.
Vinita Hotwani said…
website to setup your printer. 123hp.com/setup Get started with your new printer by downloading the software.We Support you for 123hp.com/setup Network Printer Connection which can be done by either USB or Wireless Connection.Once you switch 123hp.com/setup on your printer, on the touchscreen head over to the network section. 123hp.com/setup official website - 123.hp.com setup & complete instructions and guidance from unpacking the printer...
Vinita Hotwani said…
HBO Max, a new streaming service from WarnerMedia, is coming to Xfinity X1 and Xfinity Flex, directly hbomax.com/tvsignin alongside all of the other programming customers receive as part of their TV subscription and from our growing library of OTT partners.HBO Max is an app on my TV but, my provider gives hbomax.com/tvsignin it to me free already, but if i try to use the app on the TV either the code is not ... Hbomax.com/tvsignin.hbomax.com/tvsignin On the HBO Max sign-in screen, choose Sign In Through TV or Mobile Provider· hbomax.com/tvsignin Select Hulu as your provider and log in with your Hulu. HBO Max will arrive on Amazon Fire TV and Fire tablets on Nov. 17, six months after WarnerMedia launched the streaming service.
Vinita Hotwani said…
Avg.com/retail - Steps to Download, Install & Activate AVG Retail antivirus to safeguard your avg.com/retail computer and other devices from online Trojan risk, cyber threats.

Download norton antivirus to make your computer virus norton.com/setup free with the best support and tech team. Feel free to contact us
Vinita Hotwani said…
Redeem McAfee for downloading & installing the software and activating its subscription. Visit mcafee.com/activate for more.Visit www.mcafee.com/activate to get the latest details of subscription plans. Download mcafee.com/activate install and activate McAfee at mcafee.com/activate.
Vinita Hotwani said…
Amazon Web Services Scalable Cloud Computing Services: Audible Download Audio Books: DPReview Digital Photography: IMDb Movies, TV & Celebrities : Shopbop Designer Fashion Brands: Amazon Business Everything For Your Business: amazon.com/mytv Prime Now 2-Hour Delivery on Everyday Items: Amazon Prime Music Stream millions of songs, ad-free. Go to amazon.com/mytv and create your Amazon account, if you already have an account then sign in to your account. amazon.com/mytv Now enter the activation or verification code in the next screen that is in front of you. You are now ready to enjoy a lot of amazing and great movies and shows on your smart TV.

Easily Download 123 HP Printer Software, Quick Support for 123 HP Setup Installation, Step by Step 123hp.com/setup instruction to connect, troubleshoot, ink cartridge, etc. Ger Our First Time HP Printer Support, 123.hp.com/install, HP Printer wireless Installation Setup, Fix 123hp.com/setup printer issues.This is the website for setup your hp 123hp.com/setup printer as per the rules of 123.hp.com/setup & recover your printer in troubles too, we are the service provider through online.
Keria Helan said…
Steps to add and activate ABC on Roku
• At first, press the home button on your remote for streaming channels option.
• Search for ABC channel and choose the same.
• After you add the channel wait for the channel to download.
• Launch the channel, and you will see an ABC activation code.
• Now, open your browser on mobile or PC and visit abc.com/activate.
• Enter the activation code in the given space and click continue.
• Choose pay-tv provider's name from the list. That is, select Roku.
• After that, Sign in with your PAY TV account.
• With this, you can start the party of your favorite.
KaleemSeo said…
123movies.com official site, watch free movies, watch free movies online watch free movies online



asdasd said…
I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. david icke

asdasd said…
I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. feshop



asdasd said…
I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work.dog grooming sunny isles


asdasd said…
I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work.ku bet

asdasd said…
Fantastic Site with useful and unique content looking forward to the next update thank you.Yorkie For Sale USA

asdasd said…
Fantastic Site with useful and unique content looking forward to the next update thank you.Yorkie For Sale USA



KaleemSeo said…
Find hottest online dirty web cam models at dirtycameras.com. Sign up for free and start talking with dirty web cam girls tonight. Registering is easy and fast and you are ready to chat!"diry cam

Lisa said…
If your system isn't booting or loading applications at the same time and it takes a lot of time then in that case you first unboxed it, iolo download System Mechanic.
asdasd said…
Fantastic Site with useful and unique content looking forward to the next update thank you.nha cai

asdasd said…
Fantastic Site with useful and unique content looking forward to the next update thank you.พนัน ออนไลน์

asdasd said…
Fantastic Site with useful and unique content looking forward to the next update thank you.แทงบอลออนไลน์ Sbobet


KaleemSeo said…
We are really grateful for your blog post. You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. polyurea






asdasd said…
We stock handiest the satisfactory magic Mushroom. Therefore you can make certain the quality quality from our shop.Buy Psilocybin Mushrooms Online




vikas said…

Thanks for sharing a great article post.
iec consultants
트럼프트위터 예비군일정 스포츠토토 사이트 프리미어리그 손혜원기부 추천놀이터 코드 로투스배팅 오래된 토토사이트 사설토토 놀이터추천 안전공원
I recommend a Toto site that has been verified for safe use. 토토사이트 If you ask VIP Toto, I promise to recommend a safety playground. 안전토토사이트We promise to recommend the Toto site that is safer than anyone else, 토토 and please contact VIP Toto for a major safety playground that you can trust and use. 스포츠토토
Watton said…
Mushrooms Micro-dosing USA

We have put together a collection of psychedelics from different parts of USA and test them to make up a good list of products for our clients to consume.  Legal Psychedelics Online, where to buy shrooms online, Buy Psychedelics Online, where to buy magic mushrooms, and many more. Order Psychedelics from us online to acquire quality, legal and excellent products in the lives of our clients for a better and stress-free lifestyle. It is to be noted that we have a very discreet and secure end-to-end delivery system combined with the best growers of Legal Psychedelics Online USA, We are typically the most trusted connects of Psychedelic brands medication worldwide. We guarantee 100 percent client satisfaction and Money Back Guarantee. buy shrooms from us at very affordable prices. where to buy magic mushrooms and a magic mushroom grow kit.
For more information visit here : https://psychedelicrangers.com

We are a group of young enthusiasts who are out to blow the whistle on the magical healing powers of Psychedelics and all its other advantages by making this healing drugs available to everyone. We have experienced the powers of Psychedelics and we which to share this experience with everyone out there. We will try our best to help anyone in need. All our products have been tested and verified by all competent authorities
Our blog post - Psychedelicrangers
Contact us - Psychedelicrangers
FREQUENTLY ASKED QUESTIONS - Psychedelicrangers
Our Products - Psychedelicrangers
shipping return policy - Psychedelicrangers
Psychedelic Rangers Shop - Psychedelicrangers
10G Top Quality DMT Crystal | Spirit Molecule | N,N DMT
28G Dutch Champagne MDMA Crystal
Akaslot said…
เล่นสล็อตแล้วก็เกมแจ็คพอตที่น่าเร้าใจที่ AKASLOT
ตรวจสอบคอลเลกชั่น pgslot แล้วก็เกมที่มากมายรวมทั้งนานัปการของพวกเรา คุณจะศึกษาและทำการค้นพบภาพที่สวยสดงดงาม การเล่นเกมที่สมจริงสมจัง รวมทั้งช่องทางที่จะชนะรางวัลแจ็คพอตสุดดีเลิศ ตัวเลือกไม่จบไม่สิ้น ตั้งแต่เกมเริ่มแรกของ Grosvenor Casinos รวมทั้ง Tiki Runner แล้วก็ Atlantis Megaways ไปจนกระทั่งเกมใหม่แล้วก็พิเศษสุดตื่นเต้น ตัวอย่างเช่น pgslot คุณมั่นใจว่าจะได้เจอกับเกมที่จะทำให้ท่านเพลินใจ
เพลิดเพลินใจกับลักษณะพิเศษตั้งแต่ฟรีสปินและก็เกมโบนัสไปจนกระทั่งแจ็คพอตรายวันและก็การแลกเปลี่ยนเครื่องหมาย ตลอดจนการชำระเงินสูงสุด 250,000 ปอนด์สำหรับเกมที่ไม่ใช่แจ็คพอตแค่นั้น รับข้อมูลทั้งผองเกี่ยวกับสล็อตแจ็คพอตและก็เกมที่มีให้ที่ pgslot จาก AKASLOT
เล่นสล็อตออนไลน์ที่ AKASLOT
ถ้าเกิดคุณพึ่งสมัครสมาชิกกับ AKASLOT จังหวะที่คุณกำลังอยากได้เล่นสล็อตออนไลน์หรือ pgslot ด้วยต้นแบบที่เรียบง่ายแล้วก็คุณลักษณะที่เบิกบาน pgslot ออนไลน์มอบประสบการณ์การเล่นเกมที่บรรเทาและก็บีบคั้นต่ำ มีเกมให้เลือกเยอะแยะพร้อมธีมมากมายตั้งแต่อียิปต์โบราณแล้วก็ Lucky Leprechauns ไปจนกระทั่งภาพยนตร์ฮอลลีวูดรวมทั้งรายการทีวี แต่ละเกมมีข้อเสนอแนะของตนเอง ด้วยเหตุดังกล่าวแม้คุณจะไม่เคยเล่นสล็อตออนไลน์มาก่อน คุณก็จะเข้าดวงใจได้อย่างเร็วว่าควรจะมองหาเครื่องหมายใดและก็ช่องชำระเงินปฏิบัติงานเช่นไร pgslot ออนไลน์เยอะๆมีรอบหมุนฟรีซึ่งคุณสามารถหมุนวงล้อโดยไม่ต้องพนันเงินสดแล้วก็คุณลักษณะที่น่าตื่นตาตื่นใจที่ให้ท่านมีโอกาสอย่างมากขึ้นที่จะชนะรางวัลเงินสด

แนวทางเล่นสล็อตออนไลน์
ฉะนั้นคุณเล่น pgslot ออนไลน์เช่นไร เกมสล็อตแทบจะทั้งหมดมีข้อแนะนำเบื้องต้นเช่นกัน เพียงแค่คลิกปุ่มหมุนแล้วเกมจะจัดแจงที่เหลือ
เกม pgslot ไม่ต้องการที่จะอยากความชำนาญหรือแผนการใดๆก็ตามเพราะว่าความมีชัยขึ้นกับจังหวะ สล็อตออนไลน์ก็เลยล่อใจผู้เล่นทุกระดับ

ช่องชำระเงินเกมสล็อต
เพื่อชนะ คุณจำเป็นต้องจับคู่เครื่องหมายที่แบบเดียวกันในลำดับเฉพาะบนวงล้อ นี่เรียกว่าเพย์ไลน์หรือวินไลน์ แต่ละเกมจะมีปริมาณช่องชำระเงินที่แตกต่าง คุณสามารถมองว่าสิ่งพวกนี้เป็นอย่างไรในแต่ละรายการอาหารช่วยเหลือในเกม เพย์ไลน์ชอบ:
เป็นระเบียบเรียบร้อย
ในแบบอย่างซิกข์แซก
เป็นรูปตัววี
แนวเฉียง

เครื่องหมายที่ชนะในแต่ละเกมจะต่างๆนาๆ รวมทั้งช่องชำระเงินจะมีน้ำหนักต่างๆนาๆในแต่ละสปิน เมื่อคุณเล่นด้วยเงินจริง ยอดเงินของคุณจะถูกหักครั้งใดก็ตามคุณหมุนวงล้อ คุณจำเป็นต้องตกลงใจว่าจะวางเดิมพันในแต่ละเพย์ไลน์มากมายน้อยเท่าใดเพื่อประกอบเป็นเงินที่ใช้ในการเดิมพันทั้งหมดทั้งปวงของคุณ
ซึ่งแน่ๆว่าการเล่นเกมสล็อตหรือ pgslot ออนไลน์ สำหรับช่องบางช่องมิได้ใช้ช่องชำระเงินรวมทั้งชำระเงินตามปริมาณเครื่องหมายที่คุณวางบนวงล้อแทน ด้วยเกมจำพวกนี้ คุณสามารถมีแนวทางชนะมากยิ่งกว่า 2,000 แนวทาง เกม Megaways เสนอความมากมายหลายเยอะขึ้นเรื่อยๆด้วยสูงถึง 117,649 แนวทางสำหรับในการชนะ!
AnzalnaSeo said…
It’s super webpage, I was looking for something like thisThe Weeknd White Bat Country
Unknown said…
Thanks so much for sharing this great info! I am looking forward to see more posts! 안전토토사이트
Unknown said…
web-site proprietors should take this web site as an model, very clean and wonderful user friendly style 안전한놀이터
Unknown said…
I am constantly browsing online for ideas that can benefit me. Thanks! 토토검증업체
Anonymous said…
we present the verification criteria. Cool. I'll be back every day. 보증업체
Anonymous said…
I can grow up with. Please continue to do well and always do well.Some information about medicine 먹튀검증.
Anonymous said…
Thank you for sharing your thoughts. I really appreciate your Every time you have this place like this, 먹튀검증
ophunter.net said…
My brother suggested I might like this website.He was totally right. This post truly made my day.

This entry was posted in 휴게텔
massage.blue said…
You can't imagine simply how much time I had spent for this info! Thanks! 스포츠마사지
gunma.top said…
I just could not depart your site prior to suggesting that I really enjoyed the standard info a person provide for your visitors? Is going to be back often to check up on new posts 출장마사지
Hehehe said…
Hi there, I simply hopped over in your website by way of StumbleUpon. Now not one thing I’d typically learn, but I favored your emotions none the less. Thank you for making something worth reading. 먹튀검증업체 먹튀검증업체
bamgosoocom said…
I am sure this article has touched all the internet viewers, its really really
pleasant piece of writing on building up new website.
풀싸롱

Wonderful illustrated information. I thank you about that. No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject!
Nice post. I learn something new and challenging on websites I stumbleupon on a daily basis.
토토 said…
I think this is a really good article. You make this information interesting and engaging. You give readers a lot to think about and I appreciate that kind of writing. 카지노사이트
토토 said…
Thanks for your marvelous posting! I actually enjoyed reading it, you could be
a great author.I will remember to bookmark your blog and will
eventually come back from now on. I want to encourage you to continue your great 룰렛
thebestsitehere said…
You're so interesting! I don't suppose I've read anything like this before.
So great to find someone with a few unique thoughts on this issue.
Seriously.. many thanks for starting this up. This site is something that is needed on
the internet, someone with some originality!
오피사이트

Good day! I could have sworn I've been to this website
before but after reading through some of the post I realized it's new
to me. Nonetheless, I'm definitely glad I found it and I'll be bookmarking and checking
back frequently!
안전공원 said…
You should take part in a contest for one of the composed article, if just all bloggers offered the same substance as you, the web would be an obviously better place 스포츠사이트검증
안전공원 said…
먹튀사이트 your work and wondering how you managed this blog so well. It’s so remarkable thato through this valuable information whenever
안전공원 said…
You're so interesting! I don't suppose I've read through something like this before. So good to discover another person 놀이터검증업체
opbestcom8 said…
The Gaming Club Casino also offers auction services to members, a sign-up bonus and cash
gratuities available to players and club members alike.
After the player receiving this bonus, these casinos
offer other bonuses for future deposits. These
players will be provided incentives for upcoming deposits that happen to be their casino accounts.오피사이트


Anonymous said…
I can't believe there's a post like this Wow, great blog article 토토사이트보증업체
Anonymous said…
Please write good things from now on 검증사이트목록 You’re incredible! Thank you!
바카라사이트 I'm truly enjoying the design and layout οf your website.
Ιt's a very easy on the eyes which makes іt much more pleasant foor me to cоme here and visiit moге ⲟften.
카지노 Your style is so unique compared to other folks I have read stuff from.
Many thanks for posting when you’ve got the opportunity, Guess I’ll just bookmark this page.

스포츠토토 Howdy just wanted to give you a quick heads up.
The text in your post seem to be running off
the screen in Opera. I’m not sure if this is a format issue or something to do
with browser compatibility but I thought I’d post
to let you know. The design look great though! Hope you get the problem
fixed soon. Cheers

opbestcom said…

Very good written article. It will be supportive to anyone who utilizes it, including me.
Keep doing what you are doing ? can’r wait to read more posts.
very nice article.

Again, awesome web site! === 안마

(freaky)
Anonymous said…
That's amazing. I grow up watching this. Appreciate you I'll come and read every day. It's really nice to have a place like this 토토
Anonymous said…
that will help me grow 먹튀폴리스 but I believe it will help a lot in my country
Anonymous said…
my own blog and would like to find out where u got this from. I live in a different country than you 먹튀검증사이트
Unknown said…
This is exceptionally instructive substance and composed well for a change. It's토토
Unknown said…
안전놀이터 how do you say it? Relevant!! Finally I've found something which helped me. Thank you!
Unknown said…
i appreciate you and hopping for some more 메이저사이트 informative posts
Unknown said…
You made some good points there. I did a Google search about the topic and found most people will believe your blog. 먹튀검증


unknown said…
Hello, I am one of the most impressed people in your article. 안전놀이터추천 I'm very curious about how you write such a good article. Are you an expert on this subject? I think so. Thank you again for allowing me to read these posts, and have a nice day today. Thank you.


mik said…
Hurrah, that’s what I was looking for, what a information! present here at this web site, thanks admin of this site.
buy oxycontin 10mg online USA
yahanvideonet said…
Greetings! Very useful advice within this article! It’s the little changes that will make the biggest changes. Thanks for sharing! 한국야동
yahanvideonet said…
I have to thank you for the efforts you’ve put in penning this blog. I am hoping to check out the same high-grade content from you in the future as well 중국야동넷
oncaday said…

Hi there mates, how is all, and what you would like to say concerning this
article, in my view its truly awesome in favor of me.

Look at my web-site 카지노커뮤니티


(IVY)
PGSLOT said…
นักเดิมพันท่านใดสนใจเกมที่ดีเล่นง่ายเข้าใจง่ายเชิญมาทางนี้ Flirting Scholar เกมสล็อต รูปแบบ 5 รีล 3 แถว เป็นอีกเกมที่จะทำให้ทุกท่านสนุกและเพลิดเพลินจนลืมเวลามาพร้อมโปรโมชั่นดีๆ และภายในเว็บไซต์เกมสล็อตออนไลน์ของเรานั้นยังมีเกมอีกมากมายให้ทุกท่านได้เลือกร่วมสนุก ไม่เพียงเท่านั้น เรายังมีระบบ ฝาก-ถอน อัตโนมัติที่รวดเร็วและปลอดภัยให้ทุกท่านได้รับความสะดวกสบายกับการใช้บริการจากเว็บไซต์ขงอเราอีกด้วยs
UnKnown said…
What a post I've been looking for! I'm very happy to finally read this post. 토토사이트 Thank you very much. Can I refer to your post on my website? Your post touched me a lot and helped me a lot. If you have any questions, please visit my site and read what kind of posts I am posting. I am sure it will be interesting.
unknown said…
I'm so happy to finally find a post with what I want. 메이저토토사이트 You have inspired me a lot. If you are satisfied, please visit my website and leave your feedback.


Unknown said…
I would like to bring this to the notice of the public about how I came in contact with Mr Pedro after I lost my job and was denied a loan by my bank and other financial institutions due to my credit score. I could not pay my children's fees. I was behind on bills, about to be thrown out of the house due to my inability to pay my rent, It was during this period my kids were taken from me by foster care. Then I set out to seek funds online where I lost $3,670 that I borrowed from friends which I was ripped off by two online loan companies. Until i read about Mr Pedro helping people with an loan online in which this email was stated (pedroloanss@gmail.com) somewhere on the internet, Still wasn't convinced because of what i have been through until a relative of mine who is a clergy also told me about the ongoing loan scheme  at a very low interest rate of 2% and lovely repayment terms without penalty for default of payment. I had no choice than to also contact them, which I did. Mr Pedro responded back to me. That day was the best and greatest day of my life which can never be forgotten when I received a credit alert for the $400,000.00 Usd loan amount I applied for. I utilized the loan effectively to pay up my debts and to start up a business and today my kids and I are so happy and fulfilled. You can also contact them through email: (pedroloanss@gmail.com / WhatsApp Text : +18632310632)   Why am I doing this? I am doing this to save as many that are in need of a loan not to be victims of scams on the internet. Thanks and God bless you all, I'm Oleksander Artem from Horizon Park BC , Ukrain.
oncaday34com said…
I am genuinely thankful to the holder of this site who has shared this enormous piece of writing at here.
카지노사이트


>cc
Dung said…
I figure this article can be enhanced a tad. There are a couple of things that are dangerous here, and if you somehow managed to change these things, this article could wind up a standout amongst your best ones. I have a few thoughts with respect to how you can change these things. 먹튀검증업체


Michael Pere said…
That was super awesome; very good post enjoy a lot of information to read this post thanks for sharing this very good post.
Kid Cudi Merch
Kid Cudi Hoodie
Global certification

If you want to get certified in the following exams.

Your success key is here 🤓

Talk to me if you are interested to save your time⏳ and money!!💵



💯% Passing Guarantee .

Payment after pass



👉Cisco- CCNA, CCNP, Specialty

👉PMI-PMP, PMI-ACP, PMI-PBA, PMI-CAPM

👉Google-Google Cloud Associate & Google Cloud Professional

👉 People Cert- ITILv4

👉 AWS- Associate, Professional, Specialty

👉 Juniper- Associate, Professional, Specialty

👉Microsoft - All exams

👉SAFe- All exams

👉Scrum- All Exams

👉Azure

👉CompTIA - All exams

👉CISM/CISA/CRISC

👉VMWARE-ALL

👉SCRUM-ALL

👉GRE

👉GMAT

👉TOFEL

👉ORACLE-ALL & many more...



Get in touch for more discount offer 😉

http://wa.me/+ 1 (414) 310 9953

Whatsapp or call us

Email: globalcertificationscenter@gmail.com
koko said…
หากวันนี้มองหาหนทางการเอาตัวรอดจากยุคเศรษฐกิจย่ำแย่เช่นนี้วันนี้เราขอเสนอทางออกที่ไร้ความเสี่ยงเล่นเลยที่ค่าย PG ออโต้ แจกเครดิตฟรีหนึ่งร้อยเปอร์เซ็นต์สำหรับสมาชิกใหม่ทุกคนมีโบนัสแจกกระจายจ่ายให้ไม่มีอั้น
SITES 2021 said…
This page really has all the information and facts I wanted about this subject and didn’t know who to ask
고스톱
SITES 2021 said…
I tell everyone how amazing you are. You have the best laugh.
Wonderful article! That is the kind of info that are supposed to be shared around the internet.
I used to be able to find good info from your
blog posts.
일본야동
SITES 2021 said…
I precisely wished to appreciate you again. I’m not certain what I would’ve created in the absence of these thoughts shared by you over my situation. It was before a very hard scenario in my opinion, nevertheless coming across your professional manner you treated it forced me to cry with joy. I’m just happy for this advice as well as sincerely hope you comprehend what an amazing job you’re putting in educating many others by way of a site. More than likely you haven’t met all of us.
토토사이트
Anonymous said…
พบกับความสุขและเงินมากมายกันได้แล้วที่ค่าย PG Auto ค่ายที่ไม่เคยทำให้ใครผิดหวังตอบสนองความต้องการของท่านอย่างไร้ขีดจำกัดเล่นเลยที่ www.pgslot.sexy เงินมากมายที่มีมาเพื่อท่านแจกเครดิตฟรีทุกวันมีโบนัสให้ทุกนาทีคุ้มกว่านี้ไม่มีอีกแล้ว
경마사이트 said…

Its like you read my mind! You appear to grasp so much approximately this, such as you wrote the ebook in it or something. I believe that you just could do with some p.c. to force the message home a little bit, however instead of that, that is great blog. An excellent read. I will certainly be back.
국내경마

magosucowep
Voworlds said…

You can now easily login for a favorite online casino
and click on the mobile tab. Not everybody is able to find
a very good site as a way to make use of the cheap deals and entertainment though.
Nowadays gaming websites provides best features and functionalities
that assure an interactive platform. 오피월드

오피월드 said…
Good morning!! I am also blogging with you. In my blog, articles related to are mainly written, and they are usually called . If you are curious about , please visit!!
오피월드

oworldsmewep

오피월드 said…
I’m not sure exactly why but this weblog is loading incredibly slow for me. Is anyone else having this problem or is it a problem on my end? I’ll check back later on and see if the problem still exists
오피월드

oworldsmewep
sportstotolink said…
Hi there, its pleasant post about media print, we all understand media is
a wonderful source of facts 토토
roulettesitetop said…
I really appreciate this wonderful post that you have provided for us. I feel strongly that love and read more on this topic 카지노사이트
magosucom9 said…
This blog is an amazing blog, the contents here are always very educative and useful, the reason I make it my regular blog of visit so as to widen my knowledge every day, thanks for always sharing useful and interesting information, you can checkout this, it can be useful to you as well. Also visit my site :: 일본경마


JIYANG
My Site said…
I envy your piece of work, appreciate it for all the good content.
바카라사이트
토토
oncagosu said…
Your article has provoked a ton of positive hobby. I can see why since you have made such a decent showing of making it intriguing. 토토


Anonymous said…
I am not sure where your getting your information, but good topic. I needs to spend some time learning much more or understanding more. 메이저안전사이트
Anonymous said…
After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how 사설토토사이트
Anonymous said…
Thank you very much for sharing this article. It helped me a lot and made me feel a lot. Please feel free 토토검증업체
magosucomyang said…
I have read several good stuff here. Certainly worth bookmarking for revisiting. I wonder how much effort you put to make such a fantastic informative website.경마

Hahaha said…
It's so nice to know you. I hope you also welcome me.<a href="
Hahaha said…
It's so nice to know you. I hope you also welcome me.카지노사이트검증If you welcome me, please visit my blog and write. My blog is on It's hard because of Covid - 19, but let's do our best!!

NT said…
The assignment submission period was over and I was nervous, 메이저사이트추천 and I am very happy to see your post just in time and it was a great help. Thank you ! Leave your blog address below. Please visit me anytime.


james said…
Why Do Students Look For SPSS Assignment Help ? SPSS is generally used for the study of statistical data that includes many disciplines.You can take the help of our experienced SPSS project Help professionals to compose the assignment. With our SPSS assignment help, you can get more meaningful insights from your data and predict what is likely to happen.
For more info- Get Assignment Help
NT said…
I came to this site with the introduction of a friend around me and I was very impressed when I found your writing. I'll come back often after bookmarking! 메가슬롯


sportstotome said…
Pleasant learning picking up article. This post is truly the best on this significant theme. 스포츠토토
sportstotome said…
Excellent post. I simply stumbled upon your blog and also wanted to claim that I’ve really appreciated exploring your blog articles. Regardless I will be subscribing to the nourish and i also hope you write again shortly! 바카라사이트
sportstotome said…
Thanks for an interesting blog. What else may I get that sort of info written in such a perfect approach? I have an undertaking that I am just now operating on, and I have been on the lookout for such info. 카지노사이트
NOMO said…
I finally found what I was looking for! I'm so happy. 바카라사이트 Your article is what I've been looking for for a long time. I'm happy to find you like this. Could you visit my website if you have time? I'm sure you'll find a post of interest that you'll find interesting.


NOMO said…
What a post I've been looking for! I'm very happy to finally read this post. 메이저사이트추천 Thank you very much. Can I refer to your post on my website? Your post touched me a lot and helped me a lot. If you have any questions, please visit my site and read what kind of posts I am posting. I am sure it will be interesting.


Hahaha said…
Hello, I am one of the most impressed people in your article. 먹튀검증 What you wrote was very helpful to me. Thank you. Actually, I run a site similar to you. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.


NT said…
The assignment submission period was over and I was nervous, 메이저놀이터 Your posts are neatly organized with the information I want, so there are plenty of resources to reference. I bookmark this site and will find your posts frequently in the future. Thanks again ^^


I know this website presents quality depending articles or reviews and other stuff, is there any other site which gives these kinds of data in quality? 바카라사이트인포
alone said…
This is the perfect post.casino trực tuyến It helped me a lot. If you have time, I hope you come to my site and share your opinions. Have a nice day.


horsemedicare said…
Buy Dexaphenylbutazone online
www.horsemedicare.com
shasaralala said…
is one very interesting post. 메이저사이트I like the way you write and I will bookmark your blog to my favorites.


NOMO said…
Don't go past my writing! Please read my article only once. Come here and read it once"카지노사이트


NOMO said…
Hello, I am one of the most impressed people in your article. 우리카지노 What you wrote was very helpful to me. Thank you. Actually, I run a site similar to you. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.


johnwhat said…

Thank you because you have been willing to share information with us. we will always appreciate all you have done here because I know you are very concerned with quialty content.
tupac clothing
토토사이트 said…
Your explanation is organized very easy to understand!!! I understood at once. Could you please post about 우리카지노?? Please!!


Anonymous said…
Please post useful information often. We will continue to visit in the future. Have a good day. 토토사이트
Anonymous said…
is still among the leading topics of our time. I appreciate your post and look forward to more. 토토검증
cer said…
Se você está enfrentando problemas financeiros hoje, recomendamos o Ninja Vs Samurai , um jogo que o ajudará a resolver todos os seus problemas financeiros. More brgabe. Invista conosco, mas vale a pena. Não há risco para este investimento. Bom e vale mais do que isto, não mais. Aberto para que você jogue 24 horas por dia.
Studytonight said…
Hi there, thank you for sharing such great informative post with us. It is really helpful.
Windows error code
Kernel_Security_Check_Error
ERR_CONNECTION_RESET" Error
ePSXe Black Screen problem
Slot said…
ตามมาเยอะ
ambbet
Digital Mania said…
In the todays world everybody wants to be rich in his life so we are providing you various tips how to become rich overnight?
johnwhat said…
I appreciate your willingness to share information with us. Since I know you care very much about quality content, I will always appreciate all you have done.
FOG Essentials
T-Shirts
Essentials Hoodies
Sweatshirts
Jackets
Sweatpants
Shorts
Shoes
Unknown said…
This blog is so nice to me. I will keep on coming here again and again.
buy cheap codeine 30mg online
keonhacai said…
This comment has been removed by the author.
Zonahobisaya said…
Bra nettside : Zonahobisaya
Bra nettside : Zonahobisaya
Bra nettside : Zonahobisaya
Bra nettside : Zonahobisaya
Bra nettside : Zonahobisaya
Bra nettside : Terluas
Bra nettside : Resep
Bra nettside : Sinopsis Film
Thanks for an interesting blog. What else may I get that sort of info written in such a perfect approach? I have an undertaking that I am just now operating on, and I have been on the lookout for such info 먹튀검증 It's amazing. I want to learn your writing skills. In fact, I also have a website. If you are okay, please visit once and leave your opinion. Thank you.


Chrome Heart said…
I have read all articles of your website .There are very informative for me. Thanks for sharing.
ravensky867 said…
unlimited trick ।। 바둑이사이트 킹스 홀덤 다운로드 ab wining trick ।। new teen patti vungo unlimited winning trick 2021 ।।
unknown said…
Article scroll
Nice post Keep it up! Digital Marketing Company in USA We give the best Services for online marketing for your business.
our services we provide SEO, PPC, Web designing, etc.
Bunny .cow said…
รับสิทธิประโยชน์ดี ๆ ได้มากมายกับ SAGAME โปรโมชั่น 50 รับ 100 โปรโมชั่นบาคาร่าที่ตอบโจทย์สายปั่นมากที่สุด ซึ่งเว็บเกมของเราพร้อมแจกหนักจัดเต็มไม่อั้น และสามารถเลือกรับโปรโมชั่นอื่น ๆ บนหน้าเว็บเกมของเราได้ตลอด 24 ชั่วโมง เป็นเว็บเดียวในไทยที่มีโปรโมชั่นบาคาร่ามากที่สุด และแจกหนักจัดเต็มกว่าใครแน่นอน สามารถกดรับสิทธิประโยชน์ดี ๆ แบบนี้ได้ทั้งสมาชิกใหม่ และสมาชิกเก่า และหากผู้เล่นท่านใดชนะเดิมพัน 10 รอบบิลอย่างต่อเนื่อง รับเงินรางวัลจากเว็บเกมของเราทันที 700 บาท และมีกิจกรรมดี ๆ ให้ร่วมสนุกลุ้นรับของรางวัลอีกมากมาย แจกฟรีทุกเดือน เราพร้อมมอบสิ่งที่ดีที่สุดให้กับคุณ
hoddies said…
This comment has been removed by the author.
massage said…
Guys just sharing, I've found this interesting! Check it out!

Massage in Berlin |
Steven Smith said…
Student stress isn’t merely a passing discomfort; it’s a pervasive issue that can significantly affect academic performance, mental health, and overall quality of life. When faced with overwhelming academic stress, many students have assignments done by Do My Assignment Online writing service. When students are overwhelmed by stress, it can lead to burnout, anxiety, depression, and even physical health problems. Furthermore, stress can hinder cognitive functions, making it difficult for students to concentrate, learn, and retain information effectively. Therefore, addressing student stress is not just a matter of personal well-being but also crucial for ensuring the success and development of future generations.

James William said…
Thanks so much for sharing, this great knowledge blog keep it up good work.
Barbara Nimmo said…
Your insights on post-modern C tooling in your latest blog post were truly fascinating. It's refreshing to delve into the evolving landscape of programming languages and tools.
Seatgeek Promo Code $40 Off
Ashlee Rolfson said…
Your blog post on "Post-Modern C Tooling" is a fascinating exploration into the evolving landscape of programming tools. It's always exciting to see how technology adapts and transforms.
frugalishness
Kit Charles said…
"A well-deserved round of applause for your consistent excellence in blogging. Your posts are a perfect blend of education and entertainment, making them both informative and enjoyable. Thank you for sharing your expertise with us!"
chena hot springs resort shuttle coupons
Abigale Huels said…
"I've recommended your blog to my friends, and they are as impressed as I am. Your ability to capture the essence of diverse subjects and present them in an engaging manner is truly remarkable. Keep up the fantastic work!"
KFC Student Discount

Popular posts from this blog

Draft 3 of, ^Let's write a unit test!^

Is PostgreSQL good enough?