[PATCH 0/9] test/py: Various enhancements for lab use
From: Simon Glass <sjg@chromium.org> This series provides several improvements associated with the sjg labgrid integration: - restart boards reliably during tests - console logging to a file - allow putting Labgrid into verbose mode - easier debugging of test.py Simon Glass (9): test/py: Provide an option to abort on an exception hooks: Pass the verbose flag to labgrid test/py: Fix a spurious tab in get_details() hooks: Allow writing the console log to a file sandbox: Drop expect_reset from restart_uboot_with_flags() test/py: Create a new function for EFI capsule updates test/py: Create a function to restart a board test/py: hooks: Provide a proper way to restart U-Boot pcduino: Enable EFI self-tests configs/Linksprite_pcDuino3_defconfig | 1 + test/hooks/bin/console.labgrid-sjg | 2 +- test/hooks/bin/ellesmere/common-labgrid-sjg | 1 + test/hooks/bin/restart.labgrid-sjg | 43 +++++++++++++++++++ test/hooks/bin/u-boot-test-restart | 38 ++++++++++++++++ test/py/conftest.py | 38 ++++++++++------ test/py/console_base.py | 28 +++++++++++- test/py/console_board.py | 14 ++++++ test/py/console_sandbox.py | 7 +-- .../test_capsule_firmware_fit.py | 10 ++++- .../test_capsule_firmware_raw.py | 5 ++- test/py/tests/test_efi_selftest.py | 2 +- test/py/tests/test_reset.py | 8 ++++ 13 files changed, 171 insertions(+), 26 deletions(-) create mode 100755 test/hooks/bin/restart.labgrid-sjg create mode 100755 test/hooks/bin/u-boot-test-restart -- 2.43.0 base-commit: 290829cc0d20dc4da5a8dfa43b94adcf368bc1b1 branch: labf
From: Simon Glass <sjg@chromium.org> It is quite tricky to debug problems in the test.py code itself, as when something goes wrong the exception failure is caught and reported as a test failure. Add a -E option to simplify debugging. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/conftest.py | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/test/py/conftest.py b/test/py/conftest.py index 4460e5a3af2..7ac777fea7f 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -87,6 +87,9 @@ def pytest_addoption(parser): help='Compile U-Boot before running tests') parser.addoption('--buildman', default=False, action='store_true', help='Use buildman to build U-Boot (assuming --build is given)') + parser.addoption( + '-E', '--allow-exceptions', '-E', default=False, action='store_true', + help='Avoid catching exceptions with test failures') parser.addoption('--gdbserver', default=None, help='Run sandbox under gdbserver. The argument is the channel '+ 'over which gdbserver should communicate, e.g. localhost:1234') @@ -336,6 +339,7 @@ def pytest_configure(config): ubconfig.connection_ok = True ubconfig.timing = config.getoption('timing') ubconfig.role = config.getoption('role') + ubconfig.allow_exceptions = config.getoption('allow_exceptions') env_vars = ( 'board_type', @@ -507,6 +511,9 @@ def ubman(request): if not ubconfig.connection_ok: pytest.skip('Cannot get target connection') return None + if ubman_fix.config.allow_exceptions: + ubman_fix.ensure_spawned() + return ubman_fix try: ubman_fix.ensure_spawned() except OSError as err: @@ -880,20 +887,23 @@ def pytest_runtest_protocol(item, nextitem): test_list.append(item.name) tests_not_run.remove(item.name) - try: + if ubman_fix.config.allow_exceptions: msg_log(msg) - except: - # If something went wrong with logging, it's better to let the test - # process continue, which may report other exceptions that triggered - # the logging issue (e.g. ubman_fix.log wasn't created). Hence, just - # squash the exception. If the test setup failed due to e.g. syntax - # error somewhere else, this won't be seen. However, once that issue - # is fixed, if this exception still exists, it will then be logged as - # part of the test's stdout. - import traceback - print('Exception occurred while logging runtest status:') - traceback.print_exc() - # FIXME: Can we force a test failure here? + else: + try: + msg_log(msg) + except: + # If something went wrong with logging, it's better to let the test + # process continue, which may report other exceptions that triggered + # the logging issue (e.g. ubman_fix.log wasn't created). Hence, just + # squash the exception. If the test setup failed due to e.g. syntax + # error somewhere else, this won't be seen. However, once that issue + # is fixed, if this exception still exists, it will then be logged + # as part of the test's stdout. + import traceback + print('Exception occurred while logging runtest status:') + traceback.print_exc() + # FIXME: Can we force a test failure here? log.end_section(item.name) -- 2.43.0
From: Simon Glass <sjg@chromium.org> The V variable includes the -v flag if verbose operation is requested in Labgrid. Add it to the console cmdline. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/hooks/bin/console.labgrid-sjg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/hooks/bin/console.labgrid-sjg b/test/hooks/bin/console.labgrid-sjg index f521acea4a5..13b3d169681 100644 --- a/test/hooks/bin/console.labgrid-sjg +++ b/test/hooks/bin/console.labgrid-sjg @@ -38,6 +38,6 @@ exec labgrid-client -V do-bootstrap ${bootstrap:-1} -V do-build ${build:-1} \ -V do-send ${send:-0} -V do-clean ${clean:-0} ${vars} \ - -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} \ + -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} ${V} \ -r "${U_BOOT_BOARD_IDENTITY}" ${strategy} ${verbose} -a console \ ${console_log} -- 2.43.0
From: Simon Glass <sjg@chromium.org> One line is indented with tab instead of space. Fix it. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/conftest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/py/conftest.py b/test/py/conftest.py index 7ac777fea7f..16f28db6e23 100644 --- a/test/py/conftest.py +++ b/test/py/conftest.py @@ -170,7 +170,7 @@ def get_details(config): if build_dir_extra: env['U_BOOT_BUILD_DIR_EXTRA'] = build_dir_extra - # Make sure the script sees that it is being run from pytest + # Make sure the script sees that it is being run from pytest env['U_BOOT_SOURCE_DIR'] = source_dir proc = subprocess.run(cmd, stdout=subprocess.PIPE, -- 2.43.0
From: Simon Glass <sjg@chromium.org> Labgrid has a --log-output option to allow writing its log output to a file. Add a way to control this from Labgrid's U-Boot scripts. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/hooks/bin/console.labgrid-sjg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/hooks/bin/console.labgrid-sjg b/test/hooks/bin/console.labgrid-sjg index 13b3d169681..fb91c5eacb8 100644 --- a/test/hooks/bin/console.labgrid-sjg +++ b/test/hooks/bin/console.labgrid-sjg @@ -38,6 +38,6 @@ exec labgrid-client -V do-bootstrap ${bootstrap:-1} -V do-build ${build:-1} \ -V do-send ${send:-0} -V do-clean ${clean:-0} ${vars} \ - -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} ${V} \ + -V process-limit ${BUILDMAN_PROCESS_LIMIT:-0} ${log_output} ${V} \ -r "${U_BOOT_BOARD_IDENTITY}" ${strategy} ${verbose} -a console \ ${console_log} -- 2.43.0
From: Simon Glass <sjg@chromium.org> No caller uses this argument, so drop it. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/console_sandbox.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/test/py/console_sandbox.py b/test/py/console_sandbox.py index da55d2fcc1f..dd721067f8f 100644 --- a/test/py/console_sandbox.py +++ b/test/py/console_sandbox.py @@ -60,14 +60,11 @@ class ConsoleSandbox(ConsoleBase): cmd += self.sandbox_flags return Spawn(cmd, cwd=self.config.source_dir, decode_signal=True) - def restart_uboot_with_flags(self, flags, expect_reset=False, use_dtb=True): + def restart_uboot_with_flags(self, flags, use_dtb=True): """Run U-Boot with the given command-line flags Args: flags: List of flags to pass, each a string - expect_reset: Boolean indication whether this boot is expected - to be reset while the 1st boot process after main boot before - prompt. False by default. use_dtb: True to use a device tree file, False to run without one Returns: @@ -77,7 +74,7 @@ class ConsoleSandbox(ConsoleBase): try: self.sandbox_flags = flags self.use_dtb = use_dtb - return self.restart_uboot(expect_reset) + return self.restart_uboot(False) finally: self.sandbox_flags = [] self.use_dtb = True -- 2.43.0
From: Simon Glass <sjg@chromium.org> EFI capsule updates cause a reboot of U-Boot on startup. This is the only case of this behaviour, so create a separate function for it, rather than having a parameter for the function that every board uses. This will make it easier to adjust how restart_uboot() works. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/console_base.py | 13 +++++++++++-- test/py/console_sandbox.py | 2 +- .../test_efi_capsule/test_capsule_firmware_fit.py | 10 ++++++++-- .../test_efi_capsule/test_capsule_firmware_raw.py | 5 ++++- 4 files changed, 24 insertions(+), 6 deletions(-) diff --git a/test/py/console_base.py b/test/py/console_base.py index 56201052283..3e8f2cda1c6 100644 --- a/test/py/console_base.py +++ b/test/py/console_base.py @@ -542,10 +542,19 @@ class ConsoleBase(object): pass self.p = None - def restart_uboot(self, expect_reset=False): + def restart_uboot(self): """Shut down and restart U-Boot.""" self.cleanup_spawn() - self.ensure_spawned(expect_reset) + self.ensure_spawned(False) + + def restart_and_expect_reset(self): + """Shut down and restart U-Boot, expecting it to reset itself! + + The reset is itself expected to cause another reset after the U-Boot + banner appears + """ + self.cleanup_spawn() + self.ensure_spawned(True) def get_spawn_output(self): """Return the start-up output from U-Boot diff --git a/test/py/console_sandbox.py b/test/py/console_sandbox.py index dd721067f8f..5430323b6f1 100644 --- a/test/py/console_sandbox.py +++ b/test/py/console_sandbox.py @@ -74,7 +74,7 @@ class ConsoleSandbox(ConsoleBase): try: self.sandbox_flags = flags self.use_dtb = use_dtb - return self.restart_uboot(False) + return self.restart_uboot() finally: self.sandbox_flags = [] self.use_dtb = True diff --git a/test/py/tests/test_efi_capsule/test_capsule_firmware_fit.py b/test/py/tests/test_efi_capsule/test_capsule_firmware_fit.py index 016274533cd..71c2c47f1f9 100644 --- a/test/py/tests/test_efi_capsule/test_capsule_firmware_fit.py +++ b/test/py/tests/test_efi_capsule/test_capsule_firmware_fit.py @@ -58,7 +58,10 @@ class TestEfiCapsuleFirmwareFit(): 'config_efi_capsule_on_disk_early') # reboot - ubman.restart_uboot(expect_reset = capsule_early) + if capsule_early: + ubman.restart_and_expect_reset() + else: + ubman.restart_uboot() with ubman.log.section('Test Case 1-b, after reboot'): if not capsule_early: @@ -92,7 +95,10 @@ class TestEfiCapsuleFirmwareFit(): 'config_efi_capsule_authenticate') # reboot - ubman.restart_uboot(expect_reset = capsule_early) + if capsule_early: + ubman.restart_and_expect_reset() + else: + ubman.restart_uboot() with ubman.log.section('Test Case 2-b, after reboot'): if not capsule_early: diff --git a/test/py/tests/test_efi_capsule/test_capsule_firmware_raw.py b/test/py/tests/test_efi_capsule/test_capsule_firmware_raw.py index b8cb483b380..8f548ca4d5a 100644 --- a/test/py/tests/test_efi_capsule/test_capsule_firmware_raw.py +++ b/test/py/tests/test_efi_capsule/test_capsule_firmware_raw.py @@ -133,7 +133,10 @@ class TestEfiCapsuleFirmwareRaw: 'config_efi_capsule_authenticate') # reboot - ubman.restart_uboot(expect_reset = capsule_early) + if capsule_early: + ubman.restart_and_expect_reset() + else: + ubman.restart_uboot() with ubman.log.section('Test Case 3-b, after reboot'): if not capsule_early: -- 2.43.0
From: Simon Glass <sjg@chromium.org> Add a new restart_board() function which handles restarting the board. Provide an parameter which indicates whether the reset has already happened. Make use of this from restart_uboot() Change one test over to use this function and add a new 'reset' test for sandbox. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/console_base.py | 19 +++++++++++++++++-- test/py/tests/test_efi_selftest.py | 2 +- test/py/tests/test_reset.py | 8 ++++++++ 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/test/py/console_base.py b/test/py/console_base.py index 3e8f2cda1c6..e5f9b0e26e0 100644 --- a/test/py/console_base.py +++ b/test/py/console_base.py @@ -544,8 +544,23 @@ class ConsoleBase(object): def restart_uboot(self): """Shut down and restart U-Boot.""" - self.cleanup_spawn() - self.ensure_spawned(False) + self.restart_board() + + def restart_board(self, need_reset=True): + """Restart a board so it is ready for use + + If a reset is needed the board is re-spawned. If the reset has already + happened it checks for a reboot happening on the console + + Args: + need_reset (bool): True if a reset should be initiated, False if it + has already happened + """ + if need_reset: + self.cleanup_spawn() + self.ensure_spawned(False) + else: + self.run_command(cmd='', send_nl=False, wait_for_reboot=True) def restart_and_expect_reset(self): """Shut down and restart U-Boot, expecting it to reset itself! diff --git a/test/py/tests/test_efi_selftest.py b/test/py/tests/test_efi_selftest.py index 12cbe5caa9b..bd44e0088e8 100644 --- a/test/py/tests/test_efi_selftest.py +++ b/test/py/tests/test_efi_selftest.py @@ -58,7 +58,7 @@ def test_efi_selftest_watchdog_reboot(ubman): ubman.run_command(cmd='bootefi selftest', wait_for_prompt=False) if ubman.p.expect(['resetting', 'U-Boot']): raise Exception('Reset failed in \'watchdog reboot\' test') - ubman.run_command(cmd='', send_nl=False, wait_for_reboot=True) + ubman.restart_uboot(need_reset=False) @pytest.mark.buildconfigspec('cmd_bootefi_selftest') def test_efi_selftest_text_input(ubman): diff --git a/test/py/tests/test_reset.py b/test/py/tests/test_reset.py index af079a70664..c85c96ef690 100644 --- a/test/py/tests/test_reset.py +++ b/test/py/tests/test_reset.py @@ -59,5 +59,13 @@ def test_reset_w(ubman): setup_reset_env(ubman) ubman.run_command('reset -w', wait_for_reboot=True) + # Checks U-Boot's command-prompt functionality after reset + test_000_version.test_version(ubman) + +@pytest.mark.buildconfigspec('sandbox') +def test_reset_sandbox(ubman): + """Test the reset command for sandbox""" + ubman.restart_board(need_reset=True) + # Checks the u-boot command prompt's functionality after reset test_000_version.test_version(ubman) -- 2.43.0
From: Simon Glass <sjg@chromium.org> Board reset in pytests is pretty ad-hoc at present. The normal approach is to cause a reset to happen, or come to a point in a test where a reset is expected. Then the test looks for the U-Boot banner, indicating that the board has rebooted. This has a few problems: - Some lab boards use USB to send U-Boot to the board. Resetting may result in a hang, since U-Boot is not present after a reset - Resetting generally does a software reset, which is different to a hardware reset. For example the TPM is not reset on a software reset so the previous TPM state is still present. This means that tests which want to check early TPM-setup cannot work - To do a hardware reset, the current integration simply throws away the ConsoleExecAttach() object and created a new one. This unnecessary with the sjg Labgrid integration and it is quite inefficient, since U-Boot is built again, written to the board again, etc. - Some boards may need a signal to be provided to indicate whence the firmware should be read. If this signal happens to not be present, the wrong firmware will be started There are various other corner cases as well. To resolve this, provide a restart_board() implementation in ConsoleExecAttach which can reliably restart a board. There is one parameter, which tells it whether the reset has already been triggered. If not, it triggers one. Implement this in the sjg Labgrid integration (i.e. 'lab mode', via a new `u-boot-test-restart` hook. Enable it for ellesmere (sjg lab). This allows send-only boards like pcduino3 to handle the EFI watchdog test, for example. There is quite a bit of cleanup which could be pursured following this change, mostly calling the new function instead ubman.run_command() and improving how sandbox restarts. This can be dealt with later. Signed-off-by: Simon Glass <sjg@chromium.org> --- test/hooks/bin/ellesmere/common-labgrid-sjg | 1 + test/hooks/bin/restart.labgrid-sjg | 43 +++++++++++++++++++++ test/hooks/bin/u-boot-test-restart | 38 ++++++++++++++++++ test/py/console_board.py | 14 +++++++ 4 files changed, 96 insertions(+) create mode 100755 test/hooks/bin/restart.labgrid-sjg create mode 100755 test/hooks/bin/u-boot-test-restart diff --git a/test/hooks/bin/ellesmere/common-labgrid-sjg b/test/hooks/bin/ellesmere/common-labgrid-sjg index 6b3eff5f277..d62f5fadf92 100755 --- a/test/hooks/bin/ellesmere/common-labgrid-sjg +++ b/test/hooks/bin/ellesmere/common-labgrid-sjg @@ -44,3 +44,4 @@ console_impl=labgrid-sjg release_impl=labgrid-sjg getrole_impl=labgrid-sjg power_impl=none +restart_impl=labgrid-sjg diff --git a/test/hooks/bin/restart.labgrid-sjg b/test/hooks/bin/restart.labgrid-sjg new file mode 100755 index 00000000000..7c533753fe3 --- /dev/null +++ b/test/hooks/bin/restart.labgrid-sjg @@ -0,0 +1,43 @@ +# Copyright 2024 Google LLC +# Written by Simon Glass +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# The variables here can come from one of two places: +# +# 1. When using the ub-xxx scripts (e.g. ub-int) they come from those scripts, +# set by the get_args.sh script +# +# 2. When running from gitlab, the variables are all empty and so take the +# default values below, except for ${strategy} which is set in the gitlab +# script + +# On input: +# strategy: Strategy arguments to use, e.g. "-s start". Normally this is +# "-s uboot -e off" but it can be "-s start -e off" or even empty +# verbose: Verbose argument to use, e.g. "-v" +# +# The 'vars' variable is set by .gitlab-ci.yml or by get_args.sh in the Labgrid +# integration + +reset="$3" +exec labgrid-client -V do-bootstrap ${bootstrap:-1} \ + -V do-send ${send:-0} -V do-reset ${reset:-0} ${vars} \ + -r "${U_BOOT_BOARD_IDENTITY}" -i uboot -s restart ${verbose} \ + --assume-ready query diff --git a/test/hooks/bin/u-boot-test-restart b/test/hooks/bin/u-boot-test-restart new file mode 100755 index 00000000000..2f60bddab2b --- /dev/null +++ b/test/hooks/bin/u-boot-test-restart @@ -0,0 +1,38 @@ +#!/bin/bash + +# Copyright 2025 Simon Glass <sjg@chromium.org> +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +# Restarts the board, bringing it back to the U-Boot prompt ready for running +# tests + +# This is called from restart_board() in test/py + +# Arguments: +# - board 'type' (U-Boot board name) +# - board identity (Labgrid role) +# - <reset_flag> +# +# <reset_flag> is 1 if this script should reset the board, 0 (or empty) if it +# has already been done + +. "$(dirname $0)/u-boot-test-common" + +. "${bin_dir}/restart.${restart_impl}" diff --git a/test/py/console_board.py b/test/py/console_board.py index bacb1e2526c..1c7ee8be7d9 100644 --- a/test/py/console_board.py +++ b/test/py/console_board.py @@ -74,6 +74,20 @@ class ConsoleExecAttach(ConsoleBase): return s + def restart_board(self, need_reset=True): + if not self.lab_mode: + super().restart_board(need_reset) + return + + self.log.action(f'Restarting board (need_reset {need_reset})') + args = [self.config.board_type, self.config.board_identity] + if need_reset: + args.append('1') + cmd = ['u-boot-test-restart'] + args + runner = self.log.get_runner(cmd[0], sys.stdout) + runner.run(cmd) + runner.close() + def close(self): super().close() -- 2.43.0
From: Simon Glass <sjg@chromium.org> The watchdog-reset test can function correctly now, so enable it on this board, as an example. Signed-off-by: Simon Glass <sjg@chromium.org> --- configs/Linksprite_pcDuino3_defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/configs/Linksprite_pcDuino3_defconfig b/configs/Linksprite_pcDuino3_defconfig index 7db10e685bc..45a2ce19853 100644 --- a/configs/Linksprite_pcDuino3_defconfig +++ b/configs/Linksprite_pcDuino3_defconfig @@ -8,6 +8,7 @@ CONFIG_DRAM_ZQ=122 CONFIG_AHCI=y # CONFIG_SYS_MALLOC_CLEAR_ON_INIT is not set CONFIG_SPL_I2C=y +CONFIG_CMD_BOOTEFI_SELFTEST=y CONFIG_SCSI_AHCI=y CONFIG_SYS_64BIT_LBA=y CONFIG_SYS_I2C_MVTWSI=y -- 2.43.0
participants (1)
-
Simon Glass