22

Is there, somewhere, a freely usable/accessible script, source file, or whatever, that is able to measure the compliance of a given C++ compiler?

For example, the Acid3 test for browsers: http://acid3.acidtests.org/

The results I dream of would be a global percentage note (or multiple notes, one for each standard, e.g. , c++98, c++11, c++14, etc.), and then detailed tests with "success" or "failure" for each of them.

Background: I had a discussion at work about boost and some challenged compilers. My interlocutor spoke about boost being an academic project, because it won't work in major C++ compilers, and me answering that mentally challenged compilers should not count. Being able to measure with code the actual conformance of a compiler would help both in evaluating the compiler, and discovering the "corner cases" that should be avoided in cross-platform code compiled with them.

Edit: 2013-06-22

Not an answer, but apparently, the C++ committee is working on the subject:

SG10, Feature Test: Clark Nelson (Intel). Investigation into whether and how to standardize a way for portable code to check whether a particular C++ product implements a feature yet, as we continue to extend the standard.

Source: http://isocpp.org/std/the-committee

Community
  • 1
paercebal
  • 1,457
  • 1
  • 11
  • 17
  • There's this: http://www.peren.com/pages/products_set.htm. Won't attempt an answer, as I haven't used it and I'm not sure of it's status in the C++ community. – yannis May 28 '13 at 21:52
  • What major compilers does Boost not work on? It works on Microsoft's Visual Studio Compiler, GCC, and CLANG. – Travis Pessetto May 28 '13 at 21:56
  • @YannisRizos : Thanks for the comment, but this is a commercial product whose results are not freely accessible. I'll rephrase my question to ask for "free/open" tests. – paercebal May 28 '13 at 21:57
  • 2
    Boost works in major C++ compilers. Perhaps not ancient versions of those compilers, and perhaps not in a comparatively obscure compilers your company is relying on, but just look at that huge list of tested compilers in the release notes. And that's just the ones the Boost guys tested themselves! Boost is in no sense of the word academic, and you don't need to to evaluate compilers to dismiss that statement. –  May 28 '13 at 21:58
  • @delnan: Thanks for the link... :-) ... Not to point fingers, but... I see no sunstudio compiler, there... Let's say it's a typo. – paercebal May 28 '13 at 22:13
  • I can't find anything on google for sunstudio. Is it now Oracle Solaris Studio? – Travis Pessetto May 28 '13 at 22:27
  • 4
    I nearly fell off my chair laughing at this question.... Its a very valid question, I am interested to here the answers..... It's 2013 - surely something as fundamental as our tool chains use TDD - or is it a case of "Do what I say, not what I do...... " – mattnz May 29 '13 at 04:01
  • 7
    Why the close votes? I believe this is a very valid and relevant question. Proof of compliance to standards is mandatory for most tools used by most professionals. – mattnz May 29 '13 at 21:16
  • 1
    @mattnz The problem is not whether there are compliance tests, but whether there are open/standardized tests. I would bet quite a bit that Microsoft/Intel/Etc. have TONS of tests for standards compliance, we just never see them, and they most certainly aren't standardized. – Chewy Gumball Jun 22 '13 at 06:08
  • 1
    I think this is one of the greatest reoccurring blunders/failures of the C++ committee. I am amazed a lot of really smart people keep doing the same dumb thing over and over. –  Jul 25 '14 at 08:38

3 Answers3

7

One way to test compliance of a given C++ compiler is to run a (large) testsuite, i.e. a bunch of C++ files that test all aspects of the language.

Well known testsuites are the gcc and llvm testsuites. These test all kinds of aspects of the compiler, obviously including conformance. However it is hard to particularly assess conformance specifically as (to the best of my knowledge) there are no special reports generated on conformance.

In this realm you have:

  • the GCC C-torture tests used to test the gcc compiler
  • the LLVM testsuite: a mix of directed tests for the clang compiler and llvm backend plus a set of full-fledged applications to test the compiler.
  • the LLVM libcxx test (part of the libcxx repository) to test C++11 support.

There are some extensive testsuites that to test C/C++ conformance available. Alas these are all commercial licenses in the order of 10k-40k.

  • SuperTest: The most extensive of this list. Roughly 3 million directed tests towards C99, C++, EmbeddedC, DSP-C conformance.
  • Plum Hall. C and C++, including the C++ library. Includes ANSI C conformance, random program generator, test templating, and a scripting interpretor to control the overall testing process.
  • Perennial. C and C++.
  • Nullstone. C only.
Martijn Rutten
  • 179
4

The closest thing I'm aware of is in the context of a specific standard version. Similarly, there are blog posts that contain like content. However, I'm not aware of a de facto or de jure body, test, or test suite that rules on such matters. Even isocpp.org doesn't appear to have conformance resources currently. The closest you can get is probably each vendor's specific conformance charts which is inconsistent at best.

e.g. Clang, GCC, MSVC, Intel

There are really only a few competitive compilers (at least in the x86/x64 space - I assume it's also the case for less popular platforms), but I agree that it would be very nice to have a reference, especially now that the standard is picking up speed.

Joel
  • 1,169
0

I am unaware of any standards testing, though the following URL is the official standards website comitee http://www.open-std.org/jtc1/sc22/wg21/.

Also, many things that are found in Boost make their way into the C++ standard. According to Dr. Kenneth Sundberg of Utah State University.

I also can't verify the accredibility of it yet, I am still trying to figure out if it is credible, but the Google Group at https://groups.google.com/forum/?fromgroups#!forum/comp.std.c++ claims that no official test exist.

Travis Pessetto
  • 247