Skip to content

Autograder does not work with parameterized tests. #3

Open
#9
Open
Autograder does not work with parameterized tests.#3
#9

Description

@nelsonlim
nelsonlim
opened on Sep 12, 2024

The autograder will produce key errors when running test files that have parameterized tests in them.

Here's an example of a parameterized test.

@pytest.mark.parametrize("guess, answer, expected",
                          [(5, 1, 'Lower'), # Lower
                           (6, 10, 'Higher')]) # Higher
def test_hint_higher_lower(guess, answer, expected):
    """Tests hint_higher_lower."""
    # GIVEN: The guess is 5 and the answer is 1
    # WHEN: hint_higher_lower(5,1) is run
    # THEN: "Lower" is returned
    # repeat for Higher.
    result = higherlower.hint_higher_lower(guess, answer)
    assert expected == result, (f"\nThe Test Failed."
                                f"\nHere's What was Expected:\n{expected}"
                                f"\nHere's What Your Function Returned:\n{result}")

I've attached a sample error and log archive below.

Run classroom-resources/autograding-python-grader@v1
/usr/bin/docker run --name cc4[6](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:7)2fe147022c32b94e38b0f6ce90d4e42922_c34c56 --label cc462f --workdir /github/workspace --rm -e "INPUT_TIMEOUT" -e "INPUT_MAX-SCORE" -e "INPUT_SETUP-COMMAND" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/opt/test-runner/bin/run.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/higher-lower-Manurox/higher-lower-Manurox":"/github/workspace" cc462f:e147022c32b94e38b0f6ce90d4e42922  "--timeout=10" "--max-score=1" "--setup-command=sudo -H pip3 install pytest"
/opt/test-runner/bin/run.sh: 1: eval: sudo: not found
TIMEOUT is 600 seconds
MAX_SCORE is 1
Running setup command: sudo -H pip3 install pytest
============================= test session starts ==============================
platform linux -- Python 3.11.2, pytest-[7](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:8).2.2, pluggy-1.5.0
rootdir: /github/workspace, configfile: pyproject.toml
plugins: subtests-0.11.0
5
collected 11 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 1[8](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:9)2, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 334, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 667, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/test-runner/runner/__init__.py", line 50, in pytest_collection_modifyitems
INTERNALERROR>     items.sort(key=_sort_by_lineno)
INTERNALERROR>   File "/opt/test-runner/runner/__init__.py", line 48, in _sort_by_lineno
INTERNALERROR>     return TestOrder.lineno(test_id, source)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/test-runner/runner/sort.py", line [9](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:10)4, in lineno
INTERNALERROR>     return cls._cache[test_id].lineno
INTERNALERROR>            ~~~~~~~~~~^^^^^^^^^
INTERNALERROR> KeyError: 'tests/test_higherlower.py::test_hint_higher_lower[5-1-Lower]'

============================ no tests ran in 0.02s =============================

logs_28192357931.zip

Activity

  1. nelsonlim
    changed the title [-]Autograder does not work with parametrized tests.[/-] [+]Autograder does not work with parameterized tests.[/+] on Sep 12, 2024

  2. samuelm-LCWC commented on Sep 16, 2024

    @samuelm-LCWC
    samuelm-LCWC
    on Sep 16, 2024

    I have the same issue persisting with this test-file, producing the same output as above.

    import pytest
from task import task

@pytest.mark.parametrize(
    "input_value, expected_output",
    [
        (10000, {"Needs": 5000, "Wants": 3000, "Savings": 2000}),
        (50000, {"Needs": 25000, "Wants": 15000, "Savings": 10000}),
        (13450, {"Needs": 6725, "Wants": 4035, "Savings": 2690}),
        (25000, {"Needs": 12500, "Wants": 7500, "Savings": 5000}),
        (347100, {"Needs": 173550, "Wants": 104130, "Savings": 69420}),
    ]
)
def test_task(input_value, expected_output):
    assert task(input_value) == expected_output

  3. sarchotic commented on Oct 11, 2024

    @sarchotic
    sarchotic
    on Oct 11, 2024

    This has also been a recurring issue for me as well

  4. emily-jia commented on Jan 31, 2025

    @emily-jia
    emily-jia
    on Jan 31, 2025

    I have the same issue too

  5. AndrewEckart commented on Apr 7, 2025

    @AndrewEckart
    AndrewEckart
    on Apr 7, 2025

    Same here!

  6. AndreFGard commented on Sep 27, 2025

    @AndreFGard
    AndreFGard
    on Sep 27, 2025

    had the same issue. Submitted a pr about it: #9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions