comparison libinterp/parse-tree/lex.h @ 16149:49dfba4fd3c5

use pure parser and reentrant lexer interfaces Making the Octave parser and lexer properly reentrant (and perhaps eventually thread safe as well) is still a work in progress. With the current set of changes the parser and lexer still use many global variables, so these changes alone do NOT make the Octave parser reentrant unless you take care to properly save and restore (typically with an unwind_protect object) relevant global values before and after calling the parser. Even if global variables are properly saved and restored, the parser will NOT be thread safe. * lex.ll: Use %option reentrant an %option bison-bridge. (yylval): Delete macro. (YY_EXTRA_TYPE, curr_lexer): New macros. Undefine curr_lexer (YY_FATAL_ERROR): Update decl for reentrant scanner. (lexical_feedback::reset): Update call to yyrestart for reentrant scanner interface. (lexical_feedback::fatal_error): Update call to yy_fatal_error for reentrant scanner interface. (lexical_feedback::text_yyinput): Update calls to yyinput and yyunput for reentrant scanner interface. (lexical_feedback::flex_yyleng): Use function interface to access yyleng. (lexical_feedback::flex_yytext): Use function interface to access yytext. (lexical_feedback::push_token, lexical_feedback::current_token): Use function interface to access yylval. * oct-parse.yy: Use %define api.pure, %parse-param, and %lex-param options. (curr_lexer): Define for syntax rules section. (scanner): New macro. * oct-parse.yy: Include oct-parse.h. (octave_lex): Declare. (yyerror): Update declaration for pure parser. * parse.h (octave_lex): Delete decl. * oct-parse.yy (octave_parser::run): Pass pointer to octave_parser object to octave_parse. * lex.ll (lexical_feedback::octave_read): Call fatal_error directly instead of using YY_FATAL_ERROR. * oct-parse.yy (parse_fcn_file): Pass line and column info for lexter to gobble_leading_whitespace. Access prep_for_script_file, prep_for_function_file, parsing_class_method, input_line_number, and current_input_column through curr_parser. * parse.h, oct-parse.yy (YY_BUFFER_STATE, create_buffer, current_buffer, switch_to_buffer, delete_buffer, clear_all_buffers): Delete. * toplev.cc (main_loop): Don't create new buffer for lexer. * input.cc (get_debug_input): Likewise. * oct-parse.yy (eval_string, parse_fcn_file): Likewise. * octave.cc (octave_initialize_interpreter): Likewise. * input.cc (get_debug_input): Likewise. * oct-parse.yy (eval_string, parse_fcn_file): Create parser as needed. * octave.cc (octave_initialize_interpreter): Likewise. * input.cc (get_debug_input): Likewise. * input.cc (input_even_hook): Allow function to run even if currently defining a function. * lex.h, lex.ll (curr_lexer): Delete global variable. * parse.h, oct-parse.yy (octave_parser::curr_lexer): New data member. (octave_parser::octave_parser): Create lexer here. (curr_parser): Delete global variable. * toplev.cc (main_loop): Don't protect global curr_lexer and curr_parser variables. * oct-parse.yy (eval_string, parse_fcn_file): Likewise. * input.cc (get_debug_input): Likewise. * lex.h, lex.ll (curr_lexer): Delete global variable. * parse.h, oct-parse.yy (CURR_LEXER): New temporary global. (octave_parser::octave_parser): Set global CURR_LEXER here. * toplev.cc (main_loop): Protect CURR_LEXER prior to constructing new parser object. * input.cc (get_debug_input): Likewise. * oct-parse.yy (eval_string, parse_fcn_file): Likewise. * lex.h, lex.ll (lexical_feedback::scanner): New data member. (lexical_feedback::init): Create it. Call yylex_set_extra to store pointer to lexical_feedback object in scanner data. (lexical_feedback::~lexical_feedback): Delete it. * lex.ll (YYG): New macro. (lexical_feedback::reset, lexical_feedback::prep_for_script_file, lexical_feedback::prep_for_function_file, lexical_feedback::process_comment, lexical_feedback::handle_close_bracket, lexical_feedback::handle_identifier, lexical_feedback::lexer_debug): Use it to access scanner data.
author John W. Eaton <jwe@octave.org>
date Wed, 27 Feb 2013 18:49:16 -0500
parents 2fd39ab12209
children 0259254a3ccc 7eb614760ddb
comparison
equal deleted inserted replaced
16148:10abbc493f50 16149:49dfba4fd3c5
24 #define octave_lex_h 1 24 #define octave_lex_h 1
25 25
26 #include <list> 26 #include <list>
27 #include <set> 27 #include <set>
28 #include <stack> 28 #include <stack>
29
30 // FIXME -- these input buffer things should be members of a
31 // parser input stream class.
32
33 typedef struct yy_buffer_state *YY_BUFFER_STATE;
34
35 // Associate a buffer with a new file to read.
36 extern OCTINTERP_API YY_BUFFER_STATE create_buffer (FILE *f);
37
38 // Report the current buffer.
39 extern OCTINTERP_API YY_BUFFER_STATE current_buffer (void);
40
41 // Connect to new buffer buffer.
42 extern OCTINTERP_API void switch_to_buffer (YY_BUFFER_STATE buf);
43
44 // Delete a buffer.
45 extern OCTINTERP_API void delete_buffer (YY_BUFFER_STATE buf);
46
47 extern OCTINTERP_API void clear_all_buffers (void);
48 29
49 extern OCTINTERP_API void cleanup_parser (void); 30 extern OCTINTERP_API void cleanup_parser (void);
50 31
51 // Is the given string a keyword? 32 // Is the given string a keyword?
52 extern bool is_keyword (const std::string& s); 33 extern bool is_keyword (const std::string& s);
171 152
172 std::stack<int> context; 153 std::stack<int> context;
173 }; 154 };
174 155
175 lexical_feedback (void) 156 lexical_feedback (void)
176 : convert_spaces_to_comma (true), do_comma_insert (false), 157 : scanner (0), convert_spaces_to_comma (true),
177 at_beginning_of_statement (true), 158 do_comma_insert (false), at_beginning_of_statement (true),
178 looking_at_anon_fcn_args (false), looking_at_return_list (false), 159 looking_at_anon_fcn_args (false), looking_at_return_list (false),
179 looking_at_parameter_list (false), looking_at_decl_list (false), 160 looking_at_parameter_list (false), looking_at_decl_list (false),
180 looking_at_initializer_expression (false), 161 looking_at_initializer_expression (false),
181 looking_at_matrix_or_assign_lhs (false), 162 looking_at_matrix_or_assign_lhs (false),
182 looking_for_object_index (false), 163 looking_for_object_index (false),
193 init (); 174 init ();
194 } 175 }
195 176
196 ~lexical_feedback (void); 177 ~lexical_feedback (void);
197 178
198 void init (void) 179 void init (void);
199 {
200 // The closest paren, brace, or bracket nesting is not an object
201 // index.
202 looking_at_object_index.push_front (false);
203 }
204 180
205 void reset (void); 181 void reset (void);
206 182
207 void prep_for_script_file (void); 183 void prep_for_script_file (void);
208 184
294 void display_token (int tok); 270 void display_token (int tok);
295 271
296 void fatal_error (const char *msg); 272 void fatal_error (const char *msg);
297 273
298 void lexer_debug (const char *pattern, const char *text); 274 void lexer_debug (const char *pattern, const char *text);
275
276 // Internal state of the flex-generated lexer.
277 void *scanner;
299 278
300 // TRUE means that we should convert spaces to a comma inside a 279 // TRUE means that we should convert spaces to a comma inside a
301 // matrix definition. 280 // matrix definition.
302 bool convert_spaces_to_comma; 281 bool convert_spaces_to_comma;
303 282
405 lexical_feedback (const lexical_feedback&); 384 lexical_feedback (const lexical_feedback&);
406 385
407 lexical_feedback& operator = (const lexical_feedback&); 386 lexical_feedback& operator = (const lexical_feedback&);
408 }; 387 };
409 388
410 // The current state of the lexer.
411 extern lexical_feedback *curr_lexer;
412
413 #endif 389 #endif