Improve log display

Add support for toggling display of timestamps in the logs, so
the user no longer has to navigate away to the settings page
to do this. This is available on both PipelineRun and TaskRun
pages. We always request timestamps and just show / hide them
depending on the user's preference. This is persisted to browser
localStorage as with the toggle on the settings page.

Add support for detecting GitHub Actions workflow command-style
log levels in log output. This provides an improved user
experience as it allows for filtering logs to hide unwanted
noise, e.g. debug logs, by default. We may consider allowing
the format to be customised in a future release depending on
user feedback.

Refactor the styles so `LogFormat` now correctly owns most
of the styling of the log content, with `Log` only responsible
for additional styling of the container.

Refactor use of the `LogsToolbar` component to allow for
customisable use by third-party consumers of the Dashboard
components. This means they can much more easily take advantage
of the new features, such as toggling timestamps and log levels,
without having to reimplement the menu and related code themselves.

Eliminate redundant use of `split` and `join` calls when processing
the logs, improving performance. `LogFormat` now receives an
array of log line objects, pre-parsed into the new structure
with the `timestamp`, `level` (optional), and `message` fields.
Where a multiline log is encountered, the timestamp of the first
line is reused for subsequent lines in that log.

Fix issue where in some cases a blank line did not reserve vertical
space, leading to cramped display of logs. Now each line is
guaranteed to occupy a minimum height, ensuring blank lines
output in the logs to aid in readability are preserved in the UI.

Update `FormattedDate` to add support for displaying seconds, as
this is quite important in the log context. Default to `false` for
this setting so existing date / timestamps in other parts of the
UI are unaffected. The full raw timestamp as received in the logs
in displayed in a tooltip on hover.

Update unit tests to reflect the new and changed components and
behaviours.

Update common PipelineRun E2E to exercise the new log toolbar
and validate the log content is rendered as expected.

Add new stories to cover the new functionality. Update existing
stories to demonstrate use of the new functionality in context.

Update Carbon:
- resolve issue with Plex Mono font

  Some glyphs weren't included in the Plex version packaged with
  previous Carbon releases, resulting in broken formatting for some
  log content, e.g. using box characters to print tables.

  In `@carbon/react` 1.71.0 the Plex version has been updated, as well
  as changing how it's consumed. Instead of a single package with all
  of the font variants, they're now published as separate packages per
  font family. Add the `$use-per-family-plex` flag to our config to
  use these new packages. The custom `$font-path` is still required
  for compatibility with Vite.

- resolve issue with duplicate onChange events from MenuItemSelectable

- resolve issue with duplicate onChange events when clearing a ComboBox

- document the log viewer feature, the new log format, and the existing
  external logs support

Notes:
- colours of the log level badges are based on the colours of the
  Carbon `Tag` component, with their opacity reduced so they're
  not as intense due to the potentially large number of them that
  could be displayed in the logs. These all meet minimum colour
  contrast ratio required for WCAG 2.0 level AA (i.e. > 4.5:1).
- the default log level is 'info' if no log level is explicitly
  provided in the logs, however we only display the badge when
  the log level is explicitly set. This avoids unnecessary and
  unwanted noise / clutter in the logs when not using the new
  log format.
- highlight and hover state included to highlight log lines, aiding
  in consuming the content, especially with longer log lines where
  the log level badge may not be adjacent to the content being read.
  A future update to the log viewer will add support for line
  wrapping but this is out of scope for this particular change.

Author: AlanGreene

.eslintrc.cjs

@@ -65,7 +65,10 @@ module.exports = {
     'import/no-extraneous-dependencies': 'off',
     'import/no-named-as-default': 'off',
     'import/no-named-as-default-member': 'off',
-    'import/no-unresolved': ['error', { ignore: ['\\.svg\\?react$'] }],
+    'import/no-unresolved': [
+      'error',
+      { ignore: ['\\.svg\\?react$', '\\.txt\\?raw$'] }
+    ],
     'import/prefer-default-export': 'off',
     'jsx-a11y/anchor-is-valid': 'off',
     'no-case-declarations': 'off',

docs/logs.md

@@ -0,0 +1,79 @@
+<!--
+---
+linkTitle: "Logs"
+weight: 4
+---
+-->
+
+# Tekton Dashboard log viewer
+
+This guide describes the features and functionality of the log viewer provided by the Tekton Dashboard on the `TaskRun` and `PipelineRun` details pages.
+
+## Basic functionality
+
+The Tekton Dashboard log viewer supports ANSI colour codes and text styles, and will automatically detect URLs in log content and render them as clickable links opening in a new window.
+
+## Toolbar
+
+The toolbar diplayed in the log viewer includes a number of additional features, including:
+
+- maximize: increase the area available to the log viewer by hiding the task list and run header. This allows the user to eliminate distractions from other parts of the app and focus on the log content.
+- open in new window: open the raw logs in a separate browser window. This provides an unmodified and unprocessed view of the logs, without any of the added features provided by the log viewer.
+- download: download the raw logs as a text file.
+- user preferences: these are persisted to browser local storage and applied to all logs in the app. See the following sections for more details.
+
+### Timestamps
+
+The Dashboard will always request logs with timestamps from the Kubernetes pod logs API, and show / hide the timestamps in the log viewer based on the user's preference. This can be toggled from the settings menu in the toolbar at the top of the log viewer.
+
+The timestamps are localised based on users' browser settings, with the raw timestamp value received from the API provided as a tooltip on hover.
+
+In releases prior to Tekton Dashboard v0.54, the timestamp preference was found on the Settings page, and governed whether or not timestamps were requested from the Kubernetes API server.
+
+### Log levels
+
+The log viewer parses log lines to detect the associated log level and decorate them accordingly to help with log consumability. The format supported is described below.
+
+```
+<timestamp> ::<level>::<message>
+```
+
+- `timestamp` is provided by the Kubernetes API server
+- `level` is one of `error`, `warning`, `notice`, `info`, `debug`
+   - `debug` logs are hidden by default
+   - any log line without an explicit `level` is considered as `info`, but will not display the log level badge to avoid redundancy in the UI where users are not using the supported log format
+- `message` is any other content on the line, and may contain ANSI codes for formatting, etc.
+
+For example, the following snippet would output a log line at the `warning` level:
+
+```sh
+echo '::warning::Something that may require attention but is non-blocking…'
+```
+
+The displayed log levels can be changed via the settings menu in the toolbar at the top of the log viewer.
+
+## Logs persistence
+
+By default, Tekton Dashboard loads the logs from the Kubernetes API server, using the pod logs API. However, it also supports loading logs from an external source when the container logs or the pods associated with the `TaskRuns` are no longer available on the cluster.
+
+This functionality is described in detail, along with a full walk-through of an example configuration, in [Tekton Dashboard walk-through - Logs persistence](./walkthrough/walkthrough-logs.md).
+
+It can be enabled by providing the `--external-logs` flag to the installer script, or configured directly in the Dashboard deployment's args.
+
+When configured, the Dashboard will first attempt to load pod logs normally, and if they're unavailable will fallback to the provided external logs service by making a `GET` request to the provided endpoint with the following format:
+
+```
+GET <external-logs>/<namespace>/<podName>/<container>?startTime=<stepStartTime>&completionTime=<stepCompletionTime>
+```
+
+- `namespace`: the namespace containing the run
+- `podName`: the name of the `Pod` resource associated with the selected `TaskRun`
+- `container`: the name of the container associated with the selected `step`
+- `stepStartTime`: the start time of the step container
+- `stepCompletionTime`: the completion time of the step container
+
+If the start / completion times are unavailable their respective query parameters will be omitted from the request.
+
+---
+
+Except as otherwise noted, the content of this page is licensed under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/). Code samples are licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

package-lock.json

@@ -18,9 +18,7 @@ import {
 
 export default function ActionableNotification(props) {
   return (
-    <FeatureFlags
-      flags={{ 'enable-experimental-focus-wrap-without-sentinels': true }}
-    >
+    <FeatureFlags enableExperimentalFocusWrapWithoutSentinels>
       <CarbonActionableNotification {...props} />
     </FeatureFlags>
   );

packages/components/src/components/ActionableNotification/ActionableNotification.jsx

@@ -16,6 +16,7 @@ import { FormattedDate, FormattedRelativeTime, useIntl } from 'react-intl';
 const FormattedDateWrapper = ({
   date,
   formatTooltip = formattedDate => formattedDate,
+  includeSeconds = false,
   relative
 }) => {
   const intl = useIntl();
@@ -47,6 +48,7 @@ const FormattedDateWrapper = ({
         year={yearFormat}
         hour="numeric"
         minute="numeric"
+        {...(includeSeconds ? { second: 'numeric' } : null)}
       />
     );
   }
@@ -56,7 +58,8 @@ const FormattedDateWrapper = ({
     month: 'long',
     year: 'numeric',
     hour: 'numeric',
-    minute: 'numeric'
+    minute: 'numeric',
+    ...(includeSeconds ? { second: 'numeric' } : null)
   });
   formattedDate = formatTooltip(formattedDate);
   return <span title={formattedDate}>{content}</span>;

packages/components/src/components/FormattedDate/FormattedDate.jsx

@@ -33,3 +33,9 @@ export const Relative = {
 };
 
 export const Absolute = {};
+
+export const Seconds = {
+  args: {
+    includeSeconds: true
+  }
+};

packages/components/src/components/FormattedDate/FormattedDate.stories.js

@@ -21,8 +21,18 @@ describe('FormattedDate', () => {
   });
 
   it('handles absolute date formatting', () => {
-    const { queryByText } = render(<FormattedDate date="2019/12/01" />);
-    expect(queryByText(/Dec 1, 2019/i)).toBeTruthy();
+    const { queryByText } = render(
+      <FormattedDate date="2019/12/01 12:13:14" />
+    );
+    expect(queryByText(/Dec 1, 2019, 12:13/i)).toBeTruthy();
+    expect(queryByText(/:14/i)).toBeFalsy();
+  });
+
+  it('handles absolute date formatting with seconds', () => {
+    const { queryByText } = render(
+      <FormattedDate date="2019/12/01 12:13:14" includeSeconds />
+    );
+    expect(queryByText(/Dec 1, 2019, 12:13:14/i)).toBeTruthy();
   });
 
   it('handles absolute date formatting for current year', () => {

packages/components/src/components/FormattedDate/FormattedDate.test.jsx

@@ -27,19 +27,16 @@ import {
 import DotSpinner from '../DotSpinner';
 import LogFormat from '../LogFormat';
 
-const LogLine = ({ data, index, style }) => (
-  <div style={style}>
-    <LogFormat>{`${data[index]}\n`}</LogFormat>
-  </div>
-);
-
-const itemSize = 15; // This should be kept in sync with the line-height in SCSS
+const itemSize = 16; // This should be kept in sync with the line-height in SCSS
 const defaultHeight = itemSize * 100 + itemSize / 2;
 
+const logFormatRegex =
+  /^((?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3,9}Z)\s?)?(::(?<level>error|warning|info|notice|debug)::)?(?<message>.*)?$/s;
+
 export class LogContainer extends Component {
   constructor(props) {
     super(props);
-    this.state = { loading: true };
+    this.state = { loading: true, logs: [] };
     this.logRef = createRef();
     this.textRef = createRef();
   }
@@ -244,7 +241,27 @@ export class LogContainer extends Component {
   };
 
   getLogList = () => {
-    const { stepStatus, intl } = this.props;
+    const {
+      intl,
+      logLevels,
+      parseLogLine = line => {
+        if (!line?.length) {
+          return { message: line };
+        }
+
+        const {
+          groups: { level, message, timestamp }
+        } = logFormatRegex.exec(line);
+        return {
+          level,
+          message,
+          timestamp
+        };
+      },
+      showLevels,
+      showTimestamps,
+      stepStatus
+    } = this.props;
     const { reason } = (stepStatus && stepStatus.terminated) || {};
     const {
       logs = [
@@ -255,8 +272,35 @@ export class LogContainer extends Component {
       ]
     } = this.state;
 
-    if (logs.length < 20000) {
-      return <LogFormat>{logs.join('\n')}</LogFormat>;
+    let previousTimestamp;
+    const parsedLogs = logs.reduce((acc, line) => {
+      const parsedLogLine = parseLogLine(line);
+      if (!parsedLogLine.timestamp) {
+        // multiline log, use same timestamp as previous line
+        parsedLogLine.timestamp = previousTimestamp;
+      } else {
+        previousTimestamp = parsedLogLine.timestamp;
+      }
+
+      if (
+        !logLevels ||
+        // we treat lines with no log level as if they specified 'info'
+        // but we don't display a default level for these lines to avoid
+        // unnecessary noise for users not using the expected log format
+        (!parsedLogLine.level && logLevels.info) ||
+        logLevels[parsedLogLine.level]
+      ) {
+        acc.push(parsedLogLine);
+      }
+      return acc;
+    }, []);
+    if (parsedLogs.length < 20_000) {
+      return (
+        <LogFormat
+          fields={{ level: showLevels, timestamp: showTimestamps }}
+          logs={parsedLogs}
+        />
+      );
     }
 
     const height = reason
@@ -266,12 +310,19 @@ export class LogContainer extends Component {
     return (
       <List
         height={height}
-        itemCount={logs.length}
-        itemData={logs}
+        itemCount={parsedLogs.length}
+        itemData={parsedLogs}
         itemSize={itemSize}
         width="100%"
       >
-        {LogLine}
+        {({ data, index, style }) => (
+          <div style={style}>
+            <LogFormat
+              fields={{ level: showLevels, timestamp: showTimestamps }}
+              logs={[data[index]]}
+            />
+          </div>
+        )}
       </List>
     );
   };
@@ -333,7 +384,7 @@ export class LogContainer extends Component {
       logs += decoder.decode(value, { stream: !done });
       this.setState({
         loading: false,
-        logs: logs.split('\n')
+        logs: logs.split(/\r?\n/)
       });
     } else {
       this.setState({
@@ -376,7 +427,7 @@ export class LogContainer extends Component {
       } else {
         this.setState({
           loading: false,
-          logs: logs ? logs.split('\n') : undefined
+          logs: logs ? logs.split(/\r?\n/) : undefined
         });
         if (continuePolling) {
           clearTimeout(this.timer);

packages/components/src/components/Log/Log.jsx

@@ -11,14 +11,16 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+import { useArgs } from '@storybook/preview-api';
+
 import Log from './Log';
 import LogsToolbar from '../LogsToolbar';
 
 const ansiLog =
   '\n=== demo-pipeline-run-1-build-skaffold-app-2mrdg-pod-59e217: build-step-git-source-skaffold-git-ml8j4 ===\n{"level":"info","ts":1553865693.943092,"logger":"fallback-logger","caller":"git-init/main.go:100","msg":"Successfully cloned https://github.com/GoogleContainerTools/skaffold @ \\"master\\" in path \\"/workspace\\""}\n\n=== demo-pipeline-run-1-build-skaffold-app-2mrdg-pod-59e217: build-step-build-and-push ===\n\u001b[36mINFO\u001b[0m[0000] Downloading base image golang:1.10.1-alpine3.7\n2019/03/29 13:21:34 No matching credentials were found, falling back on anonymous\n\u001b[36mINFO\u001b[0m[0001] Executing 0 build triggers\n\u001b[36mINFO\u001b[0m[0001] Unpacking rootfs as cmd RUN go build -o /app . requires it.\n\u001b[36mINFO\u001b[0m[0010] Taking snapshot of full filesystem...\n\u001b[36mINFO\u001b[0m[0015] Using files from context: [/workspace/examples/microservices/leeroy-app/app.go]\n\u001b[36mINFO\u001b[0m[0015] COPY app.go .\n\u001b[36mINFO\u001b[0m[0015] Taking snapshot of files...\n\u001b[36mINFO\u001b[0m[0015] RUN go build -o /app .\n\u001b[36mINFO\u001b[0m[0015] cmd: /bin/sh\n\u001b[36mINFO\u001b[0m[0015] args: [-c go build -o /app .]\n\u001b[36mINFO\u001b[0m[0016] Taking snapshot of full filesystem...\n\u001b[36mINFO\u001b[0m[0036] CMD ["./app"]\n\u001b[36mINFO\u001b[0m[0036] COPY --from=builder /app .\n\u001b[36mINFO\u001b[0m[0036] Taking snapshot of files...\nerror pushing image: failed to push to destination gcr.io/christiewilson-catfactory/leeroy-app:latest: Get https://gcr.io/v2/token?scope=repository%3Achristiewilson-catfactory%2Fleeroy-app%3Apush%2Cpull\u0026scope=repository%3Alibrary%2Falpine%3Apull\u0026service=gcr.io exit status 1\n\n=== demo-pipeline-run-1-build-skaffold-app-2mrdg-pod-59e217: nop ===\nBuild successful\n\r\r\n';
 
-const long = Array.from({ length: 60000 }, (v, i) => `Line ${i + 1}\n`).join(
-  ''
+const long = Array.from({ length: 60000 }, (v, i) => `Line ${i + 1}`).join(
+  '\n'
 );
 
 const performanceTest = Array.from(
@@ -30,7 +32,7 @@ export default {
   component: Log,
   decorators: [
     Story => (
-      <div style={{ width: '500px' }}>
+      <div style={{ width: 'auto' }}>
         <Story />
       </div>
     )
@@ -85,13 +87,17 @@ export const ANSICodes = {
 export const Windowed = {
   args: {
     fetchLogs: () => long,
+    showLevels: true,
+    showTimestamps: true,
     stepStatus: { terminated: { reason: 'Completed', exitCode: 0 } }
   }
 };
 
 export const Performance = {
   args: {
     fetchLogs: () => performanceTest,
+    showLevels: true,
+    showTimestamps: true,
     stepStatus: { terminated: { reason: 'Completed', exitCode: 0 } }
   },
   name: 'performance test (<20,000 lines with ANSI)'
@@ -109,8 +115,39 @@ export const Skipped = {
 
 export const Toolbar = {
   args: {
-    fetchLogs: () => 'A log message',
-    stepStatus: { terminated: { reason: 'Completed', exitCode: 0 } },
-    toolbar: <LogsToolbar name="step_log_filename.txt" url="/step/log/url" />
+    fetchLogs: async () =>
+      (await import('./samples/timestamps_log_levels.txt?raw')).default,
+    logLevels: {
+      error: true,
+      warning: true,
+      info: true,
+      notice: true,
+      debug: false
+    },
+    showLevels: true,
+    showTimestamps: false,
+    stepStatus: { terminated: { reason: 'Completed', exitCode: 0 } }
+  },
+  render: args => {
+    const [, updateArgs] = useArgs();
+    return (
+      <Log
+        {...args}
+        toolbar={
+          <LogsToolbar
+            logLevels={args.showLevels ? args.logLevels : null}
+            name="step_log_filename.txt"
+            onToggleLogLevel={logLevel =>
+              updateArgs({ logLevels: { ...args.logLevels, ...logLevel } })
+            }
+            onToggleShowTimestamps={showTimestamps =>
+              updateArgs({ showTimestamps })
+            }
+            showTimestamps={args.showTimestamps}
+            url="/step/log/url"
+          />
+        }
+      />
+    );
   }
 };