CS50 Python - PSet 7 Numb3rs - "test_numb3rs.py catches numb3rs.py only checking if first byte of IPv4 address is in range"

Question

I am currently working on the Problem Set 7 of CS50's Introduction to Programming with Python. We have to implement a function that expects an IPv4 address as input and checks whether this IPv4 address is valid. However, when I use Check50 to check my code and my test program I get the following result:

:) numb3rs.py exists
:) numb3rs.py prints True for 127.0.0.1
:) numb3rs.py prints True for 255.255.255.255
:) numb3rs.py prints True for 140.247.235.144
:) numb3rs.py prints False for 256.255.255.255
:) numb3rs.py prints False for 64.128.256.512
:) numb3rs.py prints False for 8.8.8
:) numb3rs.py prints False for 10.10.10.10.10
:) numb3rs.py prints False for 2001:0db8:85a3:0000:0000:8a2e:0370:7334
:) numb3rs.py prints False for cat
:( correct numb3rs.py passes all test_numb3rs.py checks
    expected exit code 0, not 1
:| **test_numb3rs.py catches numb3rs.py only checking if first byte of IPv4 address is in range**
    can't check until a frown turns upside down
:| **test_numb3rs.py catches numb3rs.py accepting expecting five-byte IPv4 address**
    can't check until a frown turns upside down

Unfortunately I cannot figure out what I'm doing wrong... My program seems to work fine, but my test program somehow does not fulfil the requirements...

Here is my code:

import re


def main():
    print(validate(input("IPv4 Address: ")))


def validate(ip):
    if re.search(
        r"^(?:1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.(?:1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.(?:1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.(?:1?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])$",
        ip,
    ):
        return "True"
    else:
        return "False"


if __name__ == "__main__":
    main()

And here is the code for my test program (for pytest):

from numb3rs import validate


def test_validate_bytes():
    assert validate("") == "False"
    assert validate(".") == "False"
    assert validate("..") == "False"
    assert validate("...") == "False"
    assert validate("....") == "False"
    assert validate("40") == "False"
    assert validate(".161") == "False"
    assert validate("205.") == "False"
    assert validate("243.29") == "False"
    assert validate(".94.200") == "False"
    assert validate("105.232.") == "False"
    assert validate("153.213.80") == "False"
    assert validate(".198.242.76") == "False"
    assert validate("143.73.13.") == "False"
    assert validate("210.8.16.106.246") == "False"


def test_validate_characters():
    assert validate("cat") == "False"
    assert validate("butterfly.bug") == "False"
    assert validate("ant.buffalo.duck") == "False"
    assert validate("chicken.barb.fox.antelope") == "False"
    assert validate("eagle.64.97.180") == "False"
    assert validate("98.donkey.55.92") == "False"
    assert validate("100.197.frog.161") == "False"
    assert validate("19.63.208.bear") == "False"


def test_validate_numbers():
    assert validate("210.379.39.176") == "False"
    assert validate("45.137.315.27") == "False"
    assert validate("188.96.152.991") == "False"
    assert validate("283.300.231.22") == "False"
    assert validate("815.22.665.241") == "False"
    assert validate("902.141.206.670") == "False"
    assert validate("111.426.448.196") == "False"
    assert validate("89.120.595.713") == "False"
    assert validate("792.657.634.218") == "False"
    assert validate("916.931.177.851") == "False"
    assert validate("784.218.415.593") == "False"
    assert validate("242.590.313.510") == "False"
    assert validate("687.313.787.677") == "False"
    assert validate("0.0.0.0") == "True"
    assert validate("255.255.255.255") == "True"
    assert validate("59.224.219.131") == "True"
    assert validate("251.61.166.78") == "True"
    assert validate("187.56.2.204") == "True"
    assert validate("201.210.110.158") == "True"
    assert validate("222.32.5.182") == "True"
    assert validate("77.203.177.126") == "True"
    assert validate("113.160.32.11") == "True"
    assert validate("1.1.1.1") == "True"

Can anyone tell me what I forgot to take care of in my test program? Thanks in advance!!!

PS: This is my first post in a public forum, so please be lenient with me ;-).

I have already tried to check for invalid numbers at various places, but Check50 claims I only check if the first byte is correct.

Check50 also claims that I don't check, if my program rejects 5 byte addresses. But I have included addresses with 5 "number blocks" in my test program.

So, I do not really get what I am doing wrong...

Answer 1

You have 2 simple errors (1 each in numb3rs.py and test_numb3rs.py). When you test with pytest, they cancel each other out, and disguise the error. Let me explain.

Your validate() function returns either "True" or "False" as STRING objects. The return needs to be a BOOLEAN, either True or False (no quotes). As a result, your test code "appears" to pass because it checks for string returns. Then, when you run check50, it also "appears" to pass the first 9 tests b/c it prints True or False. However, string objects aren't printed with quotes, so a STRING="True" and a BOOLEAN=True both print as True (no quotes).

check50 finally catches the error when it runs pytest with your test_numb3rs.py against a version of numb3rs.py known to be correct (not against your code). That's where you get the first error message:
:( correct numb3rs.py passes all test_numb3rs.py checks.
You can see more details in your browser by clicking on the link. (I highly suggest this anytime you don't understand a check50 error.) This is what mine showed:
Note: When you fix this error you should pass the last 2 tests. (They aren't checked b/c of the prior error.)

As an aside, you don't need so many tests to test the validate() function. (Mine only had 4 tests and passed. :-) That's not really enough, but you get my point.) You only need 1 test for each requirement (7 total): 1 test for valid object count (4), 2 tests for invalid object count (3,5), and 4 tests for invalid subnet values. You could add tests for subnet character values, but they are invalid just like 315 would be.

Also, each False test should only test for 1 invalid value. This is a good test:
assert validate("45.137.315.27") == False

This one has 4 invalid values, so it's not helpful. (Q: Which subnet value caused the False return? A: all 4.):
assert validate("687.313.787.677") == False