|
4 | 4 | "cell_type": "markdown",
|
5 | 5 | "metadata": {},
|
6 | 6 | "source": [
|
7 |
| - "# Regular expressions in Python" |
| 7 | + "Regular expressions are very useful in many situations, and not exclusive to Python. In fact, once you grasp the concepts, you'll find them indispensible and use them (or miss) them for many programming and data management tasks. This notebook intends to give you a flavor of the possibilities, it doesn't intend to be a comprehensive overview." |
8 | 8 | ]
|
9 | 9 | },
|
10 | 10 | {
|
11 | 11 | "cell_type": "markdown",
|
12 | 12 | "metadata": {},
|
13 | 13 | "source": [
|
14 |
| - "Regular expressions are very useful in many situations, and not exclusive to Python. In fact, once you grasp the concepts, you'll find them indispensible and use them (or miss) them for many programming and data management tasks. This notebook intends to give you a flavor of the possibilities, it doesn't intend to be a comprehensive overview." |
| 14 | + "# Requirements" |
15 | 15 | ]
|
16 | 16 | },
|
17 | 17 | {
|
|
23 | 23 | },
|
24 | 24 | {
|
25 | 25 | "cell_type": "code",
|
26 |
| - "execution_count": null, |
| 26 | + "execution_count": 4, |
27 | 27 | "metadata": {
|
28 |
| - "collapsed": true |
| 28 | + "tags": [] |
29 | 29 | },
|
30 | 30 | "outputs": [],
|
31 | 31 | "source": [
|
|
36 | 36 | "cell_type": "markdown",
|
37 | 37 | "metadata": {},
|
38 | 38 | "source": [
|
39 |
| - "## Match making" |
| 39 | + "# Match making" |
40 | 40 | ]
|
41 | 41 | },
|
42 | 42 | {
|
|
57 | 57 | "cell_type": "code",
|
58 | 58 | "execution_count": null,
|
59 | 59 | "metadata": {
|
60 |
| - "collapsed": false |
| 60 | + "collapsed": false, |
| 61 | + "jupyter": { |
| 62 | + "outputs_hidden": false |
| 63 | + } |
61 | 64 | },
|
62 | 65 | "outputs": [],
|
63 | 66 | "source": [
|
|
78 | 81 | "cell_type": "code",
|
79 | 82 | "execution_count": null,
|
80 | 83 | "metadata": {
|
81 |
| - "collapsed": false |
| 84 | + "collapsed": false, |
| 85 | + "jupyter": { |
| 86 | + "outputs_hidden": false |
| 87 | + } |
82 | 88 | },
|
83 | 89 | "outputs": [],
|
84 | 90 | "source": [
|
|
99 | 105 | "cell_type": "code",
|
100 | 106 | "execution_count": null,
|
101 | 107 | "metadata": {
|
102 |
| - "collapsed": false |
| 108 | + "collapsed": false, |
| 109 | + "jupyter": { |
| 110 | + "outputs_hidden": false |
| 111 | + } |
103 | 112 | },
|
104 | 113 | "outputs": [],
|
105 | 114 | "source": [
|
|
120 | 129 | "cell_type": "code",
|
121 | 130 | "execution_count": null,
|
122 | 131 | "metadata": {
|
123 |
| - "collapsed": false |
| 132 | + "collapsed": false, |
| 133 | + "jupyter": { |
| 134 | + "outputs_hidden": false |
| 135 | + } |
124 | 136 | },
|
125 | 137 | "outputs": [],
|
126 | 138 | "source": [
|
|
134 | 146 | "cell_type": "markdown",
|
135 | 147 | "metadata": {},
|
136 | 148 | "source": [
|
137 |
| - "## Extracting stuff" |
| 149 | + "# Extracting stuff" |
138 | 150 | ]
|
139 | 151 | },
|
140 | 152 | {
|
|
155 | 167 | "cell_type": "code",
|
156 | 168 | "execution_count": null,
|
157 | 169 | "metadata": {
|
158 |
| - "collapsed": false |
| 170 | + "collapsed": false, |
| 171 | + "jupyter": { |
| 172 | + "outputs_hidden": false |
| 173 | + } |
159 | 174 | },
|
160 | 175 | "outputs": [],
|
161 | 176 | "source": [
|
|
177 | 192 | "cell_type": "markdown",
|
178 | 193 | "metadata": {},
|
179 | 194 | "source": [
|
180 |
| - "## Substitution" |
| 195 | + "# Substitution" |
181 | 196 | ]
|
182 | 197 | },
|
183 | 198 | {
|
|
191 | 206 | "cell_type": "code",
|
192 | 207 | "execution_count": null,
|
193 | 208 | "metadata": {
|
194 |
| - "collapsed": false |
| 209 | + "collapsed": false, |
| 210 | + "jupyter": { |
| 211 | + "outputs_hidden": false |
| 212 | + } |
195 | 213 | },
|
196 | 214 | "outputs": [],
|
197 | 215 | "source": [
|
|
212 | 230 | "cell_type": "code",
|
213 | 231 | "execution_count": null,
|
214 | 232 | "metadata": {
|
215 |
| - "collapsed": false |
| 233 | + "collapsed": false, |
| 234 | + "jupyter": { |
| 235 | + "outputs_hidden": false |
| 236 | + } |
216 | 237 | },
|
217 | 238 | "outputs": [],
|
218 | 239 | "source": [
|
|
221 | 242 | " new_file_name = re.sub(r'(\\w+)_(\\d+)\\.', r'\\2_\\1.', file_name)\n",
|
222 | 243 | " print('{old:15s} -> {new}'.format(old=file_name, new=new_file_name))"
|
223 | 244 | ]
|
| 245 | + }, |
| 246 | + { |
| 247 | + "cell_type": "markdown", |
| 248 | + "metadata": {}, |
| 249 | + "source": [ |
| 250 | + "# Composition" |
| 251 | + ] |
| 252 | + }, |
| 253 | + { |
| 254 | + "cell_type": "markdown", |
| 255 | + "metadata": {}, |
| 256 | + "source": [ |
| 257 | + "Sophisticated regular expressions tend to be very hard to read. There are a couple of things you can do to mitigate that issue.\n", |
| 258 | + "* Use `re.VERBOSE` so that you can add whitespace and comments to the regular expression defintions.\n", |
| 259 | + "* Use composition, i.e., define regular expressions that describe part of the match, and compose those t match the entire expression." |
| 260 | + ] |
| 261 | + }, |
| 262 | + { |
| 263 | + "cell_type": "markdown", |
| 264 | + "metadata": {}, |
| 265 | + "source": [ |
| 266 | + "Consider the following example, a log message. We want to extract the date-time information, the log level, the process number and the exit value." |
| 267 | + ] |
| 268 | + }, |
| 269 | + { |
| 270 | + "cell_type": "code", |
| 271 | + "execution_count": 26, |
| 272 | + "metadata": {}, |
| 273 | + "outputs": [], |
| 274 | + "source": [ |
| 275 | + "log_entry = '2021-08-25 17:04:23.439405 [info]: end process 1 exited with 2'" |
| 276 | + ] |
| 277 | + }, |
| 278 | + { |
| 279 | + "cell_type": "markdown", |
| 280 | + "metadata": {}, |
| 281 | + "source": [ |
| 282 | + "Rather than writing a regular expression that describes the entire log message, we write expressions that match part of it." |
| 283 | + ] |
| 284 | + }, |
| 285 | + { |
| 286 | + "cell_type": "code", |
| 287 | + "execution_count": 33, |
| 288 | + "metadata": {}, |
| 289 | + "outputs": [], |
| 290 | + "source": [ |
| 291 | + "date = r'\\d{4}-\\d{2}-\\d{2}'\n", |
| 292 | + "time = r'\\d{2}:\\d{2}:\\d{2}\\.\\d+'" |
| 293 | + ] |
| 294 | + }, |
| 295 | + { |
| 296 | + "cell_type": "markdown", |
| 297 | + "metadata": {}, |
| 298 | + "source": [ |
| 299 | + "Let's check that the time matches." |
| 300 | + ] |
| 301 | + }, |
| 302 | + { |
| 303 | + "cell_type": "code", |
| 304 | + "execution_count": 34, |
| 305 | + "metadata": {}, |
| 306 | + "outputs": [ |
| 307 | + { |
| 308 | + "data": { |
| 309 | + "text/plain": [ |
| 310 | + "'17:04:23.439405'" |
| 311 | + ] |
| 312 | + }, |
| 313 | + "execution_count": 34, |
| 314 | + "metadata": {}, |
| 315 | + "output_type": "execute_result" |
| 316 | + } |
| 317 | + ], |
| 318 | + "source": [ |
| 319 | + "match = re.search(time, log_entry)\n", |
| 320 | + "match.group(0)" |
| 321 | + ] |
| 322 | + }, |
| 323 | + { |
| 324 | + "cell_type": "markdown", |
| 325 | + "metadata": {}, |
| 326 | + "source": [ |
| 327 | + "We cna now use `date` and `time` to match the entire date-time value." |
| 328 | + ] |
| 329 | + }, |
| 330 | + { |
| 331 | + "cell_type": "code", |
| 332 | + "execution_count": 45, |
| 333 | + "metadata": {}, |
| 334 | + "outputs": [ |
| 335 | + { |
| 336 | + "data": { |
| 337 | + "text/plain": [ |
| 338 | + "'2021-08-25 17:04:23.439405'" |
| 339 | + ] |
| 340 | + }, |
| 341 | + "execution_count": 45, |
| 342 | + "metadata": {}, |
| 343 | + "output_type": "execute_result" |
| 344 | + } |
| 345 | + ], |
| 346 | + "source": [ |
| 347 | + "regex = re.compile(r'({date}\\s+{time})'.format(date=date, time=time))\n", |
| 348 | + "match = regex.search(log_entry)\n", |
| 349 | + "match.group(1)" |
| 350 | + ] |
| 351 | + }, |
| 352 | + { |
| 353 | + "cell_type": "code", |
| 354 | + "execution_count": 46, |
| 355 | + "metadata": {}, |
| 356 | + "outputs": [], |
| 357 | + "source": [ |
| 358 | + "level = r'\\[(\\w+)\\]'\n", |
| 359 | + "msg = r'end\\s+process\\s+(\\d+)\\s+exited\\s+with\\s+(\\d+)'" |
| 360 | + ] |
| 361 | + }, |
| 362 | + { |
| 363 | + "cell_type": "code", |
| 364 | + "execution_count": 47, |
| 365 | + "metadata": {}, |
| 366 | + "outputs": [ |
| 367 | + { |
| 368 | + "name": "stdout", |
| 369 | + "output_type": "stream", |
| 370 | + "text": [ |
| 371 | + "datetime = 2021-08-25 17:04:23.439405\n", |
| 372 | + "log level: info\n", |
| 373 | + "process = 1\n", |
| 374 | + "exit status = 2\n" |
| 375 | + ] |
| 376 | + } |
| 377 | + ], |
| 378 | + "source": [ |
| 379 | + "regex = re.compile(r'({date}\\s+{time})\\s+{level}\\s*:\\s*{msg}'.format(date=date, time=time, level=level, msg=msg))\n", |
| 380 | + "match = regex.match(log_entry)\n", |
| 381 | + "print(f'datetime = {match.group(1)}')\n", |
| 382 | + "print(f'log level: {match.group(2)}')\n", |
| 383 | + "print(f'process = {match.group(3)}')\n", |
| 384 | + "print(f'exit status = {match.group(4)}')" |
| 385 | + ] |
| 386 | + }, |
| 387 | + { |
| 388 | + "cell_type": "markdown", |
| 389 | + "metadata": {}, |
| 390 | + "source": [ |
| 391 | + "Although the final regular expression is still rather long, it is easier to read and to maintain. Using `re.VERBOSE` and triple-quoted strings helps to further make the regular expression more maintainable." |
| 392 | + ] |
| 393 | + }, |
| 394 | + { |
| 395 | + "cell_type": "code", |
| 396 | + "execution_count": 48, |
| 397 | + "metadata": {}, |
| 398 | + "outputs": [ |
| 399 | + { |
| 400 | + "name": "stdout", |
| 401 | + "output_type": "stream", |
| 402 | + "text": [ |
| 403 | + "datetime = 2021-08-25 17:04:23.439405\n", |
| 404 | + "log level: info\n", |
| 405 | + "process = 1\n", |
| 406 | + "exit status = 2\n" |
| 407 | + ] |
| 408 | + } |
| 409 | + ], |
| 410 | + "source": [ |
| 411 | + "regex = re.compile(r'''\n", |
| 412 | + " ({date}\\s+{time})\\s+ # date-time, up to microsecond presision\n", |
| 413 | + " {level}\\s*:\\s* # log level of the log message\n", |
| 414 | + " {msg} # actual log message\n", |
| 415 | + " '''.format(date=date, time=time, level=level, msg=msg), re.VERBOSE)\n", |
| 416 | + "match = regex.match(log_entry)\n", |
| 417 | + "print(f'datetime = {match.group(1)}')\n", |
| 418 | + "print(f'log level: {match.group(2)}')\n", |
| 419 | + "print(f'process = {match.group(3)}')\n", |
| 420 | + "print(f'exit status = {match.group(4)}')" |
| 421 | + ] |
224 | 422 | }
|
225 | 423 | ],
|
226 | 424 | "metadata": {
|
|
239 | 437 | "name": "python",
|
240 | 438 | "nbconvert_exporter": "python",
|
241 | 439 | "pygments_lexer": "ipython3",
|
242 |
| - "version": "3.5.1" |
| 440 | + "version": "3.7.7" |
243 | 441 | }
|
244 | 442 | },
|
245 | 443 | "nbformat": 4,
|
246 |
| - "nbformat_minor": 0 |
| 444 | + "nbformat_minor": 4 |
247 | 445 | }
|
0 commit comments