Coccinelle release 1.0.0-rc1

[bpt/coccinelle.git] / changes.txt

-*- org -*-

* upcoming/planned/todo
** Language:
- Semantic patch for other languages
- Support metavariables, virtual rules and virtual identifiers
  in initialize and finalize rules
- Support for matching #ifdef, etc.
- function-typed metavariables

** Features:
- New scripting languages

** Bugfix:
- Add more information in documentation, man pages and wiki

* 1.0.0
** Language:
- Addition of initializer list metavariables.  Thanks to Michael Stefaniuc
  for noticing the need for them.
- Allow multiple position variables per token

** Features:
- Preserve spacing before // comments when not at the beginning of a line

** Bugfix:
- Corrected parsing of script strings delimited by a single quote.  Thanks
  to Eugeniy Meshcheryakov for reporting the problem.
- Improved indentation of added ifdefs.
- Allow added comments and #ifdefs to precede other added code.  Thanks to
  Jani Monoses for noticing the problem.

* 0.2.5
** Language:
- Make a very small attempt to parse C++ code, amounting to accepting
  identifiers containing ::, tilde, and template invocations.  Use the
  option -c++.  This is not likely to be very useful in practice.
- Added metavariable metavariable type.
- Add disjunctions on identifiers in some contexts
- Pretend that & indicates a pointer in -c++ mode
- Support for new and delete
- Allow arrays in smpl parameter declarations.  Thanks to Jesper Louis
  Andersen for pointing out the problem.
- Field list metavariables
- Add the ability to add __attributes__ (NB, no matching on attributes)
- Slightly improved the error message for transformation on multiple
  variable declaration.  Thanks to Jonathan Nieder for pointing out the
  problem.

** Features:
- support transformations on variables (only) in declarations that declare
  multiple variables
- allow #endif XXX in C code
- relax_include_path now applied to non local includes too, in which case
  it tries to find a unique file with a suffix of the provided name.
  this is useful for directories that are intended to be symbolic links.
- support matching and removing #undef
- support for iteration in ocaml, requires use of -no_show_diff
- calls to likely and unlikely propagate test expression status to their
  arguments
- reuse typedefs and macros from complete parsing when reparsing the
  transformed code
- better explanation when presenting glimpse tokens
- optimization for an if branch that is just { ... }
- spatch -control_flow_to_file file.c generates a file file.dot
- include files with names ending in .cpp if -c++ option is given
- removed use of the no longer supported sexplib function Conv.hashtbl_of_sexp
- add information about which Fedora packages are needed

** Bugfix:
- improved parsing of expressions to allow ... to the right of operators in
  more places
- Fix check_config for Python 2.7 on Fedora 14 (Reported-by: Michael Stefaniuc)
- Check for ocamlfind in configure (Reported-by: Paul E. McKenney)
- Postpone use of ocamlfind at runtime to report fewer errors
- Add support for Python 2.4 binding with the provided pycaml library
- Allow '@' in script code.  Thanks to Laurent Reveillere for noticing the
  problem.
- Remove an optimization of x or true to true in asttoctl2 for the case
  where x may make a modification, eg the case of -thing ?-thing
- Allow adding comments before a function.
- Introduction of newlines in function arguments needs to take into account
  that the code added by SmPL can include newlines.  Thanks to Thomas
  Gleixner for finding the problem.

* 0.2.4
** Language:
- scripts now have names and can declare metavariables that are seen by
  SmPL as identifier metavariables, see demos/{python,ocaml}tococci.cocci
- declaration (match variable declarations) and field (match structure
  field declarations) metavariables
- matching of array initializations and of enum declarations

** Features:
- compatability with Python 2.7.  Thanks to Richard W.M. Jones for pointing
  out the problem
- python and ocaml string representations of parameter lists and expression
  lists
- try simpler patterns on glimpse failure
- graceful and immediate failure when a virtual rule is defined on the
  command line, but not in the rule
- spatch returns -1 when a virtual rule is requested that is not supported
- no transformation when using * with the option -no_show_diff
- expanded +++ file name for match output
- struct *^* and enum *^* metavariable types
- allow an assignment to match a variable initialization, even if multiple
  variables are declared at the same time.
- add the option -recursive_includes
- added the option -use_idutils.  The requires the previous use of the
  script idutils_index.sh

** Bugfix:
- detect used after metavariables in simple statement metavariable match
- allow inheriting position variables over rules that make transformations
  but depend on virtual rules that are not defined on the command line
- treat top-level tokens in decl visitor.  Thanks to Peter Tummeltshammer
  for noticing a resulting problem.
- improve typing of the result of an assignment expression
- eliminate some recursive calls in C code parsing to avoid stack overflows
- better pretty printing of #define when the whole line is removed
- better pretty printing of asm code containing
  "::[input]"r"(&coherence_data[i])"
- allow pretty printing of C code with nameless bit fields
- avoid adding an error message to standard output if ocamlfind is not
  found
- struct/union/enum-typed metavariables with a metavariable type name
- better management of whitespace between deleted lines.  Thanks to
  Andriy Gapon for noticing the problem.
- improved parsing recovery for badly parsed struct/enum typedefs
- drop expanded tokens in sgrep output.  Thanks to Andriy Gapon for
  noticing the problem.
- constant strings have array type, not pointer type.  Thanks to Vasiliy
  Kulikov for noticing the problem.
- improve indentation when adding code after a function call with indented
  arguments.
- return type of sizeof converted to unsigned long.  Thanks to Vasiliy
  Kulikov for noticing the problem.
- improve spacing when adding structure initializers.  Thanks to Vasiliy
  Kulikov for noticing the problem.

* 0.2.3
** Language:
- <= constraints relating an expression metavariable to one or more
  inherited expression metavariables that it should be a subexpression of
- the -inplace and -outplace options have been renamed -in_place and
  -out_place.  -in_place no longer makes a backup.  A backup can be
  requested using the option -backup_suffix, eg -backup_suffix .bk
- identifiers can be constrained to be different from an inherited
  identifier metavariable, or from a set containing both concrete
  identifiers and inherited identifier metavariables.
- support for ocaml scripting
- ast available in ocaml scripts

** Features:
- drop inconsistent paths check when only one node was matched
- allow #define id with no definition
- attempt to add newlines in the generated code when function calls in the
  smpl cause passing column 80
- allow - in front of statement level nests, if everything inside the nest
  is also -
- 'configure' looks for some system libraries: menhirLib, sexplib, pycaml.
  Not available libraries are replaced by a copy provided with the sources.
- added static for a function goes just before what is specified in the
  semantic patch, not before any comments etc.
- Add a new option, -ignore_unknown_options, to ease the integration
  of Coccinelle as a checker in a toolchain.
- error in python code causes immediate abort of spatch
- use the same algorithm for collecting grep tokens as for collecting
  glimpse tokens
- Add scripts/spatch.bash_completion for automatic completion of common
  options under the bash shell.
- Print rule name when a script rule crashes
- Allow the declaration of an expression list or parameter list
  metavariable to specify the number of expressions or parameters,
  respectively.

** Bugfix:
- drop inconsistent paths check when only one node was matched
- better take into account virtual rules when selecting glimpse tokens
- print diff when only a comment is added
- the type of the C code !x should be int regardless of the type of x
- allow python code at the beginning of an included file
- better adjustment of whitespace when deleting adjacent lines.  Thanks to
  Wolfram Sang for pointing out the problem.
- allow metavariables to be inherited from included files
- allow ? on goto
- more graceful failure on finding a strange character in a macro parameter
  list
- support Parameter metavariables
- add space after the last comma in an added portion of an argument list
- SmPL nameless struct should only match a nameless struct, not a nameless
  union.  Thanks to Peter Tummeltshammer for pointing out the problem.
- adjustments to storage don't modify inline as well
- matching and transformation allowed on inline
- allow removed field between two ... in structure initialization
- allow removing an entire structure initialization
- manage labels within do while 0 macros
- added space after : in printing a conditional expression.  Thanks to Josh
  Triplett for noticing the problem.
- dropped trailing whitespace when code at the end of a line is deleted.
  Thanks to Josh Triplett for noticing the problem.
- Fixed environment management when script rule uses cache.  Thanks to
  Bissyande for finding the problem.
- when false matches if with no else
- ensure { ... } with whencode checks to the end of the function.  Thanks to
  Rene Rydhof Hansen for finding the problem.

* 0.2.2
** Language:
- Added ToTestExpression to iso language, see standard.iso for an
  explanation
- Added depends on to initialize and finalize script code

** Features:
- Update Emacs mode (cocci.el) to support 'virtual' rules and fix other
  keywords sush as 'depends on', 'using', 'disable'
- better treatment of != 0 in isos, communtativity for ==/!= for all
  constants
- allow adding // comments and blank lines (even after cocci + code)
- Add support for multiple -I options

** Bugfix:
- correct interaction between virtual rules and included .cocci files
- improvement in treatment of ! in isos, to avoid duplicating + code
- improvement in treatment of metavars as isos, to avoid duplicating + code
  between toplevel and variable instantiation
- test expression of smpl conditional, etc no longer assumed to have type
  int
- correct + line numbers in the patch produced when using *
- iso constant metavariable matches an identifier whose name is all capital
  letters, eg NULL
- allow / at the end of the name of a directory (-dir) or patch prefix
  (-patch)
- dropped space in + code after the binding of a type metavariable that is
  a pointer type
- better handling of . or .. in -dir name
- allow keywords and metavariable names in identifier constraints (not sure
  keywords is very useful, though)
- no lubtype on arguments of && and || in SmPL
- allow unknown as type for array indices in SmPL
- support matching of static annotation on functions that are both static
  and inline
- support ENOTDIR error in Common.lfile_exists, to allow for the case where
  an include file is in a subdirectory that exists but is an ordinary file,
  not a directory.
- better management of unbound position variables that appear in
  constraints
- cause python parser to skip over // comments, hoping that // is not
  meaningful inside python
- require + on every line of a multiline comment
- correct calculation of line numbers when there is script code
- always annotate arguments of && || and ! as test expressions in C code.

* 0.2.1
** Language:
- Add virtual identifiers
- Add coccilib.report and coccilib.trac Python modules
- coccilib.org and coccilib.report are imported by default but not loaded
  in the current scope.

** Features:
- Parse_error_msg now more helpful.  New option -verbose_parsing for even
  more information.
- Improve Python import handling. They are imported once during script
  initialization.

** Bugfix:
- correct treatment of depends on with || for virtual rules

* 0.2.0
** Language:

** Features:
- Remove duplicated code in disjunctions
- Better error message when grep finds nothing relevant.  Thanks to Joe
  Perches for the suggestion.
- added -keep_comments option for the unparsing of the transformed code
- Option "-version" now also gives information about built-in
  Python binding support.
- slightly faster environment manipulation in pathological cases
- hack added to accept well-formed #define after function header

** Bugfix:
- Proper consideration of #define macro arguments in checking for the use
  of metavariables and in computing the line numbers of complex terms
- Better parsing of included .cocci files
- Put included .cocci files in the right order
- Bind position variables only once for #include
- Fix bug in include_match that caused everything to halt when all matches
  were discarded
- Merge unlikely/likely iso rules under a iso rule named unlikely
- Some fixes to coccicheck rules, thanks to Andrew Lunn
- Support groups in regular expression, thanks to Michael Stefaniuc

* 0.1.11

** Language:
- Meta-identifier/function and constant could be filtered from SmPL by
  regular expressions using the "~=" and "!~=" operators followed by
  an OCaml regular expression (see man Str) in double quote.
- Virtual rules, which can be referenced in depends, and set and unset
  using the command-line options -D
- ++ for multiple additions

** Features:
- coccicheck: a framework to check a series of SmPL files on a project
  see scripts/coccicheck for more information

** Bugfix:
- bind position variables to the correct starting position in the case of a
  complex statement such as an if or while.  Thanks to Derek Jones for
  pointing out the problem.
- checking for non mentioned case lines in switch should be unchecked.
- space should be printed after sizeof when there are no parentheses around
  the argument.  Thanks to Daniel Walker for pointing out the problem.
- avoid introducing sharing in propagating ! over () in treatment of isos
- save_excursion has to handle and rethrow exceptions
- eliminate unnecessary consideration of CVS strings.  Thanks to David
  Young for pointing out the problem.
- completely new treatment of statement metavariables
- better type checking for macro definitions
- drop regression testing in -parse_c

* 0.1.10

** Language:
- declarations allowed in switch, suggested by Derek M. Jones

** Features:
- use interval timer for timeouts.  Thanks to Derek M. Jones for the
  implementation.
- more flexible env.sh, thanks to Derek Jones.
- faster Python invocation
- simplify unparsing in the sgrep case

** Bugfix:
- for glimpse there is no point to create a pattern containing a numeric
  constant because glimpse doesn't index them
- add spaces after commas in function calls and function headers
- made python integration more like the ubuntu version
  http://patches.ubuntu.com/by-release/extracted/debian/c/coccinelle/0.1.5dbs-2/01-system-pycaml
  this fixes some memory management problems with None, True, and False
- correct labels associated with a switch in the CFG so that a statement
  metavariable can match a switch.  Thanks to Derek Jones for pointing out
  the problem.
- keep switch pattern within switch body.  Thanks to Derek Jones for
  pointing out the problem.
- Allow expanded tokens to be stored in metavariables, as long as they are
  not removed.  But this does currently allow them in + code, which will
  produce the expansion.  Thanks to Ali-Erdem Ozcan for pointing out the
  problem.
- improved adjustment of spacing when code removed at the beginning of a line

* 0.1.9

** Language:
- allow fresh identifiers to be declared using ## such that the value mixes
  both strings and previously declared metavariables

** Features:
- better handling of expanded code containing ##. Now compute the 
  result.
- more precise warning message for the "'\' outside define".
  Thanks to Nicholas Mc Guire for pointing out the problem.
- more precise warning message related to ifdefization.
  Thanks to Derek Jones for pointing out the problem.
- we don't create any more certain files in /tmp (they may be a cause
  of security problems).
  Thanks to Eugeniy Meshcheryakov for pointing out the problem.
- More optimization for the case of just deleting a complete function.
  Allows this to happen without tracing through all the control-flow
  paths.  Thanks to Francois Bissyande for pointing out the problem.
- prevent code from being added to the beginning or end of a disjunction
- more information about why a script is not applied when using -debug option
- added -no_safe_expressions option
- added -no_loops option.  Ignores back edges derived from looping
  constructs.  This is unsafe, but perhaps useful for bug finding, as it can
  be more efficient.
- for semantic matches, allow "minus" on same code with multiple
  environments
- better error message for mismatch of parenthesis in column 0 with normal
  parenthesis.  Thanks to Derek Jones for pointing out the problem.
- allow disjunctions on function return types.  Thanks to Pierre Habouzit
  for pointing this out.

** Bugfix:
- keep disjunction in the proper order for structure initialization fields
- variables declared in different places should not seem to match each
  other
- drop complaints about label metavariables not being used
- drop test information from the type of an expression when the expression
  is bound to a metavariable
- nests should not extend beyond the before and after code, even if the
  before and after code matches the nest code
- nests should extend into conditionals that end in error exit
- take into account metavariables on "else".  Thanks to Derek Jones for
  pointing out the problem.
- print single quotes on generated character constants
- better typedef handling in the initialisation/affectation builtin
  isomorphism, cf -test init_affect_typedef
- support disjunction of types on variable declaration
- allow @ within strings in script code.  ignore // comment lines in script
  code.
- don't drop + code placed after the transformed code
- drop spaces produced by removing code before semicolons
- adjusted spacing within generated code
- less verbose -sp.  Thanks to Derek Jones for pointing out the problem.
- accept multiple type names in a SmPL typedef declaration.

* 0.1.8

** Language:

** Features:
- Metavariables now capture the cpp code contained within their definitions
- When - fragments are separated by ... or nest boundaries in the semantic
  patch, but end up matching adjacent source code, the comments, cpp code
  and whitespace that are between them are not deleted.
- better parsing of C: do expansion of macros only when needed when
  have actually a parse error and also leverage the definition of macros
  in the parsed file (or in a optional_standard.h file passed as a parameter).
  This should reduce the need for many hardcoded definitions in standard.h
- new semantics for the -macro_file option, by default now expand macros
  only when necessary. To force use the -macro_file_builtins option instead.
- a new -extract_macros command line action to help the parser. Works with
  the -macro_file option. e.g. 
    $ ./spatch -extract_macros ~/linux > /tmp/alldefs.h
    $ ./spatch -macro_file /tmp/alldefs.h -sp_file foo.cocci -dir ~/linux
- removed -D macro_file option, not consistent with what -D usually means
- reattempt to be more efficient for statement metavariables that are just
  placeholders (ie, no modification, no reuse)
- triples now returned from ctl in sorted order.  The main key is the
  state.  On the other hand, the state order does not always agree with the
  order of appearance in the code.
- spatch is now less verbose on the things it does yet handle. Less confusing
  for new users.
- slightly better error report. Thanks to Derek Jones for the suggestion.
- added the options -linux_spacing and -smpl_spacing.  -linux_spacing
  causes spatch to follow the spacing conventions of Linux, while
  -smpl_spacing causes spatch to follow the spacing in the semantic patch.
  -linux_spacing is the default.
- more informative error reporting for the already tagged token case.
  Thanks to Erik Hovland for the suggestion.

** Bugfix:
- better parsing of declare macro at toplevel and in structure. 
  cf -text xfield
- allowing back typedef names for fieldname
- better printing of else in generated code
- slightly better type inference for binary operators.
- clear out declarer names and iterator names between SmPL files (for
  -testall)
- better parsing and type checking of macro type aliases. Cf -test macro_int16.
  Thanks to Li Yingtong for pointing out the problem.
- make insert_virtual_positions tail rec, avoid stack overflow pb.
  Thanks to Diego Liziero <diegoliz@gmail.com> for pointing out the
  problem.
- Better type inference for arithmetic binary operators
  Thanks to Li Yingtong for pointing out the problem.
- Better type inference for constants
  Thanks to Li Yingtong for pointing out the problem.
- move computing of adjacency information for semantic patches to after
  application of isomorphisms, because isomorphisms can introduce "..."
- compute adjacency information for negated ...
- record with each transformation site the set of indices of the witness
  trees that caused the transformation site to come about.  Whitespace and
  comments between remove tokens associated with disjoint witness trees is
  not removed.
- correct treatment of function pointer typed parameters in the SmPL ast0
  visitor.
- better parsing error message and error recovery when comments are not ended, 
  when some macros have a weird body, and when some switch have a weird
  Body. 
  Thanks to Derek Jones for pointing out the problem.
- better detection and passing of "dangerous" ifdefs, cf -test
  double_switch.
- dropped the separation of decls and body in Seq.  This gives better
  positioning of the bindings of metavariables shared between them.
  Thanks to Erik Hovland for an example that shows the problem.

** Internals:
- supress warning in compiling ocamlsexp (warning caused by a new behavior
  of cpp used internally in processing files in ocamlsexp/)

* 0.1.7

** Language: 
- initialize and finalize script code, cf demos/initial_final.cocci

** Features:
- -iso_limit option to limit the depth of isomorphism application
- with the dir option, the include path is implicitly set to the "include"
  subdirectory of the specified directory, if the option -I is not used.
- give a seed for the name of a fresh identifier
- better handling of cpp "constructed" identifiers as in a##b, that in
  the futur will make it easier to match over those idents. 
  cf tests/pb_parsing_macro.c. Thanks to Ali-Erdem Ozcan for pointing
  out the problem. A new "parsing hack hint" is also available:
  YACFE_IDENT_BUILDER, cf standard.h.

** Bugfix:
- drop excessive "optimization" in ctl generation for while and for loops
- allow . as the name of the directory
- for type inference for an assignment, take the type of the right-hand
  side expression, not the type of the assigned variable
- allow for with a declartion in the first header element, as in C++
  (partial support)
- allow for matching against variable declarations that include only
  storage, eg static, but no type at all.
- allow for matching against types that contain both short/long and int
- allow the type metavariable in the SmPL code "unsigned T" to match a T
  that is a type consisting of more than one word, eg long int.
- -ifdef_to_if option made to process nested ifdefs
  (partial support)

** Internals: 
- improve and fix installation process (usable on BSD)
- improve and fix testing process
- apply patches from Eugeniy Meshcheryakov
- reorganize the way we parse C identifiers, especially concatenated cpp 
  identifiers as in a##b. This may lead to some regressions as we may
  not parse as much code as before.
- removed popl/ and popl09/ and popl related stuff from official distrib.

* 0.1.6

** Language:
- the ability to add comments

** Features: 
- grouping of generated rules with -hrule option
- handling of special coccinelle comments 
  /* {{coccinelle:skip_start}} */ and
  /* {{coccinelle:skip_end}} */
  allowing to give more hints to the C parser.
  Thanks to Flavien@lebarbe.net for the idea.
- the ability to print the values of more (but not all) kinds of
  metavariables from python
- new vim SmPL mode.
  Thanks to Alexander Faroy.

** Bugfix:
- consider the ident tokens also in the 2 lines before the error line for the 
  10-most-problematic-parsing-errors diagnostic.
- SmPL parser allows cast as the argument of a pointer
- SmPL type checker allows enum as an array index
- Better generation of fresh metavariables names in hrule
- no more warnings about things that should be metavariables when there is
  a disjunction in a function position
- bugfix in parser, better error message.
  Thanks to Ali-Erdem OZCAN <ali-erdem.ozcan@st.com> for the bug report.

** Internals: 

* 0.1.5

** Language: 
- added initialiser metavariable
- added sequences of designators in structures

** Features: 
- improved printing of the C code corresponding to metavariables
- improved printing when code (eg declarations) is removed at the beginning
  of a block, and then is followed by a blank line
- slightly less verbose error reporting in parsing_hacks

** Bugfix:
- fixed some problems with parsing SmPL code where a nest appears after a |
- better treatment of { }, form in macros wrt unparse_c
- less quiet for -parse_c
- improve parsing heuristics regarding macro statement

** Internals: 

* 0.1.4

** Language: 
- long long added to SmPL

** Documentation:
- add grammar reference and spatch command line options reference

** Features: 
- can match patterns of the form unsigned T or signed T, where T is a
  metavariable
- dropped the sizeof_parens isomorphism, which was made redundant by the
  paren isomorphism
- simple rule generation

** Bugfix:
- trailing , ; and ) no longer left on a line by themselves
- better treatment of error exit when the searched for code matches the
  error exit condition.
- fix incorrect treatment of parentheses in test_exps that could allow +
  code to be added twice
- don't ask whether iterator names and declarer names should be declared as
  metavariables.
- slightly better support for expression list metavariables.
- short and long allowed for array index types in SmPL
- more restrictions on type inference for pointer arithmetic in SmPL
- allow isomorphisms to apply when + code is anywhere within all - terms
- changed order of printing const and volatile
- allow  eg ... <... in plus code
- better formatting of generated if/while/etc. code
- better parse error reporting when the problem is at the end of the file

** Internals: 
- isomorphisms don't apply under signed/unsigned, to prevent the creation
  of things like unsigned signed int.  Need a better solution for this.

* 0.1.3

** Features: 
- help in building the configuration macro file. The -parse_c action
  now returns the 10 most frequent parsing errors. This give useful
  hints to extend standard.h.

** Bugfix:
- positions no longer allowed on \(, \|, and \)
- improved propagation of negation for isos in the presence of parens
- convert Todos in flow graph construction to recoverable errors
- fixed bug in treatment of when != true and when != false, to allow more
  than one of them per ...
- improve parsing of typedef of function pointer. 
- improve typing.
- parsing and typing support for old style C function declaration.
- consider position variables as modifications when optimizing the
  translation into CTL of function definitions

** Internals: 

* 0.1.2

** Bugfix:
- better handling of ifdef on statements in control flow graph.
- transform files even if they do not end in .c (thanks to Vegard Nossum)

** Internals: 
- merge code of yacfe

* 0.1.1

** Language: 
 - support for initializer at toplevel, cf -test substruct

* 0.1

** first public release of the source code:

** Language: 
 - embeded python scripting
 - position

** Features

* beta 

** first public release of the binary

* alpha

** Features
 - lots of features ... look at coccinelle research papers and tutorials.