type check with mypy as well

[tswast]

https://issues.apache.org/jira/browse/BEAM-12975 -- #976 appears to have broken Apache BEAM, which uses mypy for type checking.

This would also help with Mac developers, as I've been unable to run pytype locally due to some missing wheels.

[tswast]

Until this is added, I think we should revert the py.typed file change: #988

[tswast]

mypy output when I run locally

(dev-3.9) ➜  python-bigquery git:(issue987-mypy) mypy google/cloud
google/cloud/bigquery/retry.py:15: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/retry.py:17: error: Skipping analyzing "google.auth": found module but no type hints or library stubs
google/cloud/bigquery/retry.py:18: error: Library stubs not installed for "requests.exceptions" (or incompatible with Python 3.6)
google/cloud/bigquery/retry.py:18: error: Library stubs not installed for "requests" (or incompatible with Python 3.6)
google/cloud/bigquery/opentelemetry_tracing.py:17: error: Skipping analyzing "google.api_core.exceptions": found module but no type hints or library stubs
google/cloud/bigquery/opentelemetry_tracing.py:21: error: Cannot find implementation or library stub for module named "opentelemetry"
google/cloud/bigquery/opentelemetry_tracing.py:22: error: Cannot find implementation or library stub for module named "opentelemetry.instrumentation.utils"
google/cloud/bigquery/opentelemetry_tracing.py:23: error: Cannot find implementation or library stub for module named "opentelemetry.trace.status"
google/cloud/bigquery/magics/line_arg_parser/lexer.py:101: error: No overload variant of "AutoStrEnum" matches argument types "str", "List[Tuple[str, auto]]"
google/cloud/bigquery/magics/line_arg_parser/lexer.py:101: note: Possible overload variant:
google/cloud/bigquery/magics/line_arg_parser/lexer.py:101: note:     def AutoStrEnum(cls, o: bytes, encoding: str = ..., errors: str = ...) -> AutoStrEnum
google/cloud/bigquery/magics/line_arg_parser/lexer.py:101: note:     <1 more non-matching overload not shown>
google/cloud/__init__.py:24: error: Cannot determine type of "__path__"
google/cloud/bigquery/schema.py:118: error: Incompatible types in assignment (expression has type "int", target has type "str")
google/cloud/bigquery/schema.py:120: error: Incompatible types in assignment (expression has type "int", target has type "str")
google/cloud/bigquery/schema.py:122: error: Incompatible types in assignment (expression has type "int", target has type "str")
google/cloud/bigquery/schema.py:125: error: Incompatible types in assignment (expression has type "Optional[Dict[Any, Any]]", target has type "str")
google/cloud/bigquery/schema.py:246: error: Incompatible types in assignment (expression has type "List[Any]", target has type "str")
google/cloud/bigquery/_tqdm_helpers.py:24: error: Skipping analyzing "tqdm": found module but no type hints or library stubs
google/cloud/bigquery/_helpers.py:24: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/_helpers.py:117: error: Skipping analyzing "pyarrow": found module but no type hints or library stubs
google/cloud/bigquery/model.py:21: error: Library stubs not installed for "google.protobuf" (or incompatible with Python 3.6)
google/cloud/bigquery/model.py:23: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/model.py:24: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/external_config.py:810: error: Name "schema" already defined on line 782
google/cloud/bigquery/external_config.py:810: error: "Callable[[ExternalConfig], Any]" has no attribute "setter"
google/cloud/bigquery/external_config.py:866: error: "Type[object]" has no attribute "_RESOURCE_NAME"
google/cloud/bigquery/external_config.py:868: error: "Type[object]" has no attribute "from_api_repr"
google/cloud/bigquery/_pandas_helpers.py:24: error: Skipping analyzing "pandas": found module but no type hints or library stubs
google/cloud/bigquery/_pandas_helpers.py:32: error: Skipping analyzing "shapely.geometry.base": found module but no type hints or library stubs
google/cloud/bigquery/_pandas_helpers.py:46: error: Skipping analyzing "shapely.geos": found module but no type hints or library stubs
google/cloud/bigquery/_pandas_helpers.py:557: error: Skipping analyzing "pyarrow.parquet": found module but no type hints or library stubs
google/cloud/bigquery/_pandas_helpers.py:557: error: Skipping analyzing "pyarrow": found module but no type hints or library stubs
google/cloud/bigquery/routine/routine.py:19: error: Library stubs not installed for "google.protobuf" (or incompatible with Python 3.6)
google/cloud/bigquery/routine/routine.py:19: note: Hint: "python3 -m pip install types-protobuf"
google/cloud/bigquery/routine/routine.py:21: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/job/base.py:24: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/job/base.py:25: error: Skipping analyzing "google.api_core.future.polling": found module but no type hints or library stubs
google/cloud/bigquery/job/base.py:25: error: Skipping analyzing "google.api_core.future": found module but no type hints or library stubs
google/cloud/bigquery/job/base.py:37: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:38: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:39: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:40: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:41: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:42: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:43: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:44: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:45: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:46: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:47: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:48: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:49: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:50: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:51: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:52: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:53: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:54: error: Module has no attribute "client"
google/cloud/bigquery/job/base.py:910: error: Missing positional argument "job_type" in call to "_JobConfig"
google/cloud/bigquery/table.py:28: error: Skipping analyzing "pandas": found module but no type hints or library stubs
google/cloud/bigquery/table.py:33: error: Cannot find implementation or library stub for module named "geopandas"
google/cloud/bigquery/table.py:40: error: Skipping analyzing "shapely.geos": found module but no type hints or library stubs
google/cloud/bigquery/table.py:40: error: Skipping analyzing "shapely": found module but no type hints or library stubs
google/cloud/bigquery/table.py:47: error: Skipping analyzing "pyarrow": found module but no type hints or library stubs
google/cloud/bigquery/table.py:51: error: Skipping analyzing "google.api_core.exceptions": found module but no type hints or library stubs
google/cloud/bigquery/table.py:51: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/table.py:52: error: Skipping analyzing "google.api_core.page_iterator": found module but no type hints or library stubs
google/cloud/bigquery/table.py:54: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/table.py:197: error: Dict entry 0 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:198: error: Dict entry 1 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:199: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:345: error: Dict entry 0 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:346: error: Dict entry 1 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:347: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:348: error: Dict entry 3 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:349: error: Dict entry 4 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:350: error: Dict entry 5 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:351: error: Dict entry 6 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:352: error: Dict entry 7 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:353: error: Dict entry 8 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:354: error: Dict entry 9 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:355: error: Dict entry 10 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:356: error: Dict entry 11 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:357: error: Dict entry 12 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:359: error: Dict entry 14 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:360: error: Dict entry 15 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:361: error: Dict entry 16 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:362: error: Dict entry 17 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:363: error: Dict entry 18 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:364: error: Dict entry 19 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:365: error: Dict entry 20 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:366: error: Dict entry 21 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:367: error: Dict entry 22 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:368: error: Dict entry 23 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:369: error: Dict entry 24 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:370: error: Dict entry 25 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:371: error: Dict entry 26 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:372: error: Dict entry 27 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:373: error: Dict entry 28 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:374: error: Dict entry 29 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:375: error: Dict entry 30 has incompatible type "str": "str"; expected "str": "List[str]"
google/cloud/bigquery/table.py:810: error: Name "view_use_legacy_sql" already defined on line 808
google/cloud/bigquery/table.py:1749: error: Item "None" of "Optional[BigQueryReadClient]" has no attribute "_transport"
google/cloud/bigquery/table.py:1749: error: Item "BigQueryReadTransport" of "Union[BigQueryReadTransport, Any]" has no attribute "grpc_channel"
google/cloud/bigquery/table.py:1766: error: Incompatible default for argument "max_queue_size" (default has type "object", argument has type "int")
google/cloud/bigquery/table.py:2310: error: Incompatible types in assignment (expression has type "None", base class "object" defined the type as "Callable[[object], int]")
google/cloud/bigquery/table.py:2390: error: Incompatible types in assignment (expression has type "None", base class "object" defined the type as "Callable[[object], int]")
google/cloud/bigquery/query.py:420: error: Incompatible types in assignment (expression has type "str", target has type "Dict[str, Union[Any, str, float, Decimal, date, None]]")
google/cloud/bigquery/query.py:526: error: Return type "ArrayQueryParameter" of "from_api_repr" incompatible with return type "ScalarQueryParameter" in supertype "_AbstractQueryParameter"
google/cloud/bigquery/query.py:659: error: Return type "StructQueryParameter" of "from_api_repr" incompatible with return type "ScalarQueryParameter" in supertype "_AbstractQueryParameter"
google/cloud/bigquery/query.py:692: error: Incompatible types in assignment (expression has type "ArrayQueryParameter", variable has type "Optional[StructQueryParameter]")
google/cloud/bigquery/dataset.py:21: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/job/query.py:23: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/job/query.py:24: error: Skipping analyzing "google.api_core.future": found module but no type hints or library stubs
google/cloud/bigquery/job/query.py:25: error: Library stubs not installed for "requests" (or incompatible with Python 3.6)
google/cloud/bigquery/job/query.py:25: note: Hint: "python3 -m pip install types-requests"
google/cloud/bigquery/job/query.py:25: note: (or run "mypy --install-types" to install all missing stub packages)
google/cloud/bigquery/job/query.py:55: error: Skipping analyzing "pandas": found module but no type hints or library stubs
google/cloud/bigquery/job/query.py:56: error: Cannot find implementation or library stub for module named "geopandas"
google/cloud/bigquery/job/query.py:57: error: Skipping analyzing "pyarrow": found module but no type hints or library stubs
google/cloud/bigquery/job/query.py:142: error: "Callable[[Type[_NT], int, int, int], _NT]" has no attribute "__defaults__"
google/cloud/bigquery/job/query.py:192: error: Incompatible types in assignment (expression has type "str", variable has type "Optional[int]")
google/cloud/bigquery/job/query.py:206: error: Incompatible types in assignment (expression has type "str", variable has type "Optional[int]")
google/cloud/bigquery/job/query.py:626: error: Incompatible types in assignment (expression has type "Dict[str, Any]", variable has type "Optional[ScriptOptions]")
google/cloud/bigquery/job/query.py:1257: error: Signature of "result" incompatible with supertype "_AsyncJob"
google/cloud/bigquery/client.py:37: error: Skipping analyzing "pyarrow": found module but no type hints or library stubs
google/cloud/bigquery/client.py:44: error: Skipping analyzing "google.resumable_media.requests": found module but no type hints or library stubs
google/cloud/bigquery/client.py:47: error: Skipping analyzing "google.api_core.client_options": found module but no type hints or library stubs
google/cloud/bigquery/client.py:47: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/client.py:48: error: Skipping analyzing "google.api_core.exceptions": found module but no type hints or library stubs
google/cloud/bigquery/client.py:49: error: Skipping analyzing "google.api_core.iam": found module but no type hints or library stubs
google/cloud/bigquery/client.py:52: error: Skipping analyzing "google.cloud._helpers": found module but no type hints or library stubs
google/cloud/bigquery/client.py:54: error: Skipping analyzing "google.cloud.client": found module but no type hints or library stubs
google/cloud/bigquery/client.py:257: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:304: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:370: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:417: error: Incompatible types in assignment (expression has type "str", target has type "bool")
google/cloud/bigquery/client.py:561: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:607: error: Item "str" of "Union[str, Dataset]" has no attribute "project"
google/cloud/bigquery/client.py:609: error: Item "str" of "Union[str, Dataset]" has no attribute "to_api_repr"
google/cloud/bigquery/client.py:629: error: Item "str" of "Union[str, Dataset]" has no attribute "reference"
google/cloud/bigquery/client.py:636: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:691: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:726: error: Item "str" of "Union[str, Table, TableReference]" has no attribute "dataset_id"
google/cloud/bigquery/client.py:727: error: Item "str" of "Union[str, Table, TableReference]" has no attribute "project"
google/cloud/bigquery/client.py:728: error: Item "str" of "Union[str, Table, TableReference]" has no attribute "to_api_repr"
google/cloud/bigquery/client.py:744: error: Item "str" of "Union[str, Table, TableReference]" has no attribute "reference"
google/cloud/bigquery/client.py:744: error: Item "TableReference" of "Union[str, Table, TableReference]" has no attribute "reference"
google/cloud/bigquery/client.py:773: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:817: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:847: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:880: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:905: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:948: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:992: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1034: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1083: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Any]")
google/cloud/bigquery/client.py:1104: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1147: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Any]")
google/cloud/bigquery/client.py:1168: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1217: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Any]")
google/cloud/bigquery/client.py:1242: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1285: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Any]")
google/cloud/bigquery/client.py:1308: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1353: error: Item "str" of "Union[Dataset, DatasetReference, str]" has no attribute "path"
google/cloud/bigquery/client.py:1385: error: Incompatible default for argument "timeout" (default has type "None", argument has type "float")
google/cloud/bigquery/client.py:1429: error: Item "str" of "Union[Dataset, DatasetReference, str]" has no attribute "path"
google/cloud/bigquery/client.py:1462: note: (Skipping most remaining errors due to unresolved imports or missing stubs; fix these first)
google/cloud/bigquery/_http.py:20: error: Module "google.cloud" has no attribute "_http"
google/cloud/bigquery/magics/magics.py:159: error: Skipping analyzing "IPython": found module but no type hints or library stubs
google/cloud/bigquery/magics/magics.py:161: error: Skipping analyzing "IPython.core": found module but no type hints or library stubs
google/cloud/bigquery/magics/magics.py:165: error: Skipping analyzing "google.api_core": found module but no type hints or library stubs
google/cloud/bigquery/magics/magics.py:167: error: Skipping analyzing "google.api_core.exceptions": found module but no type hints or library stubs
google/cloud/bigquery/magics/magics.py:168: error: Skipping analyzing "google.auth": found module but no type hints or library stubs
google/cloud/bigquery/magics/magics.py:787: error: Skipping analyzing "google.api_core.gapic_v1": found module but no type hints or library stubs
google/cloud/bigquery/dbapi/cursor.py:34: error: Skipping analyzing "google.cloud.exceptions": found module but no type hints or library stubs
google/cloud/bigquery/dbapi/cursor.py:34: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 168 errors in 20 files (checked 49 source files)

[plamut]

Considering the popularity of mypy, this makes sense, although I do hope there won't be an explosion of different type checkers in the future, each with its own quirks and edge cases. Maintaining type hints to appease them all could become burdensome.

[tswast]

@loferris I think this might be a good way to get familiar with the google-cloud-bigquery client as a whole.

[tswast]

Smaller sub-task. Check our code samples and system tests with mypy (after hacking in a py.typed file). That should be sufficient for adding py.typed file in released package.

It'd still be good if the whole package passed mypy, though.

[plamut]

Is this currently being worked on? I have some free cycles this week and can look at this.

[tswast]

@plamut I don't believe it is. That'd be great!

I wonder if we get it working on main, but add the py.typed file to the v3 branch just so we communicate we might break folks whose type checkers all of the sudden start doing something with our package?

[plamut]

@tswast You mean making it work (on main), syncing the v3 branch with it, but then only add py.typed on the v3 branch? So that folks can then use the v3 branch for previews?

It sounds better than suddenly introducing py.typed in a non-major release. By only shipping it with 3.0.0 and mentioning that in the migration guide, people will have time to prepare. Their workflows won't break (assuming they correctly pin google-cloud-bigquery to < 3.0), and they will be expecting breaking changes in a new major version anyway.

Sounds like a plan to me!

[tswast]

Yes, that's what I meant.